AI document extraction that survives a bad scan.

Document extraction that keeps working on bad scans depends far more on what surrounds the model than on the model itself. Accuracy holds up on phone photographs, bundled PDFs and stamped invoices only when the pipeline repairs the image first, checks every value against arithmetic and known records, and sends doubtful fields to a person with the evidence beside them.

The pattern is familiar. You upload a clean PDF, the fields come back correct, and the room is convinced. Then the system meets the documents the business actually receives: a photograph of a stapled invoice, taken at an angle, with a coffee ring on the total.

Why does document extraction fail on real documents?

Document extraction fails on real documents because demo files are clean and real intake is not. The model that read a crisp PDF perfectly is now reading a skewed photograph with a shadow across one corner, and most of the errors that follow come from the image rather than from any failure to understand language.

Test sets are usually built from whatever was easy to find: exported PDFs and the supplier who sends tidy invoices. Real intake is whatever arrives. Across the document pipelines we have built, the same handful of physical conditions account for most failures.

None of this is unusual; it is what a finance inbox looks like on a Tuesday. A pipeline designed for the demo file treats each of these as an unexpected error. A pipeline designed for reality treats them as the input specification, which moves the effort away from prompts and towards image handling, business rules and review design.

The work before the model

Most of the accuracy recovered in a real deployment is recovered before the extraction model sees anything. This is unglamorous image and document handling, cheap to run and easy to test on its own.

How do you fix a skewed or rotated scan?

You fix a skewed scan by measuring the angle of the text lines and rotating the image until they are level. Page edges are a poor guide, because the edge is often the thing that is crooked: a photograph crops the paper unevenly, and a sheet fed at an angle sits inside a straight scanner frame.

Phone photographs need a perspective correction first. The camera was not directly above the page, so the sheet appears as a trapezium; detecting its four corners and remapping them to a rectangle flattens it, and deskewing handles the tilt that remains. Pages that bend near a staple or fold need dewarping, which straightens curved lines of text.

Orientation is a separate check. A page turned sideways or upside down is common in a hand-scanned batch and quietly ruins a run, so test each page in each orientation and keep the one that produces readable text. Many OCR engines include an orientation step for this reason.

Contrast and resolution

Normalise contrast region by region instead of across the whole page. A photograph lit from one side is bright on one edge and dim on the other, and a single adjustment for the whole page cannot suit both. Adaptive contrast methods adjust each small area against its own surroundings, so a shadow in one corner does not spoil the rest.

Colour helps too. Stamps are often in red or blue ink, and separating the colour channels can lift a stamp away from black print before anything is read, while keeping it as a separate layer for later.

Resolution is the condition preprocessing cannot fully repair. A common guideline for OCR is around 300 dots per inch for ordinary text, and fax-quality or heavily compressed files often fall well short. Upscaling cannot recover detail that was never captured, so for a source that keeps sending unreadable files, asking for a better copy at intake beats any processing trick.

If a person cannot read the total at a glance, fix the image before you touch the prompt.

Splitting bundled files and removing duplicates

Split multi-document files on layout changes rather than page count. An invoice, its delivery note and a compliment slip often arrive as one PDF with no separator, and a pipeline that assumes one document per file merges them into a single wrong record. Label each page as the start of a document or a continuation, using signals such as a new supplier header, a new document number or page numbering that restarts, and send pages that cannot be placed to review.

Duplicates arrive just as often: the same invoice emailed, posted, then forwarded by a colleague. A file fingerprint catches exact copies, but a scan and a PDF of the same invoice produce different fingerprints. Compare supplier, document number, date and total against documents already processed, and treat a match as a question for review rather than a deletion.

Keep the original file untouched beside every processed version. When a value is disputed, the question is always what the document actually said.

Should you use OCR or a vision language model?

Use whichever your documents demand, and on messy intake the answer is often both. Classic OCR (optical character recognition, software that turns an image of text into characters with a position for each word) is fast, cheap and predictable. Vision-capable language models read the page image directly and understand layout and context, which helps with irregular documents but brings a different kind of error.

The failure modes matter more than headline accuracy. When OCR fails it often fails visibly, returning a total such as 1,2O0.00 with a letter O in it, which a format check catches. A language model struggling with a blurred region can return a perfectly formatted amount that is not on the page, which is far harder to catch and is why validation is essential.

Using both together

A common production pattern runs both. The model receives the page image, often with the OCR text alongside, and returns structured fields, and each value is then matched back to the OCR words to find its position. That gives the review screen its crop, although a truly independent second read needs a pass that never saw the OCR text.

Model calls usually cost more than OCR, so run the cheap steps first: preprocessing, OCR, duplicate detection and identifying the document type. Send a page to the model only when it needs one, and store results so nothing is processed twice. Our guide to keeping model costs down covers the same thinking in more detail.

Schema-first extraction

Schema-first extraction means deciding exactly which fields you need, what each one means and what format it takes, before writing a prompt. The schema is the contract with everything downstream, and much of what looks like model error is really ambiguity in the definition.

Writing field definitions

Every field needs a definition precise enough that a new member of the finance team would pick the same value. Invoice date is a good test: an invoice can show an issue date, a tax point date, a delivery date and a due date, and a model asked for the date will choose one. Which one you meant belongs in the definition.

A good definition covers:

• Meaning. Which printed value the field refers to, and which similar values it does not.
• Format. Dates as year, month and day; amounts as a plain number with a separate currency code.
• Absence. What to return when the field is missing, which should be empty rather than a guess.
• Ambiguity. How to read a date such as 03/04, usually from the supplier's country.
• Scope. Whether an amount includes tax, discounts or delivery.

Store each value twice. The printed text serves review and disputes, and the normalised value is what the accounting system receives.

Structured output

Where the provider supports it, use structured output, a mode that constrains the model's response to a defined schema such as JSON with named fields and types. It removes responses that cannot be parsed, misnamed fields and amounts written as sentences. Enforcement varies by provider, so validate every response in your own code too, and remember that a response matching the schema can still hold the wrong total.

How do you check an extracted value is right?

You check an extracted value by testing it against facts the model did not produce: the document's own arithmetic, the records you already hold and what is plausible for that supplier. These checks are ordinary code, cheap to run and easy to explain to an auditor.

Arithmetic

For invoices, receipts and statements, the strongest signal is arithmetic. Line items should sum to the subtotal, and subtotal plus tax should equal the total. When they do, the amounts are very likely right, because independent misreads rarely cancel out; when they do not, something is wrong however confident each field looked.

Compare within a small tolerance, such as a penny or a cent, because tax rounded line by line can differ slightly from tax calculated on the subtotal. Arithmetic says nothing about the supplier or the invoice number, which is what the next checks are for.

Cross-referencing known records

Cross-referencing does much of the rest: a supplier name that closely matches the supplier list, a purchase order number that exists and is still open, bank details that match the ones on file. That last check matters beyond accuracy, because changed bank details on a familiar invoice are a well-known sign of payment fraud and should always reach a person.

These checks need reliable access to the systems that hold supplier records and purchase orders, and that access is often the hardest part of the project. Connecting them properly is data integration work and deserves its own plan.

Plausibility of dates and amounts

Plausibility checks catch errors that pass the arithmetic: an invoice dated in the future, a total far outside the supplier's usual range, a due date before the issue date, a currency the supplier has never used. None of these proves an error, but each is a good reason for a person to look.

checks:
  sum(line_items.amount)  == subtotal        tolerance 0.01
  subtotal + tax          == total           tolerance 0.01
  supplier                matches known_suppliers
  po_number               exists in open_purchase_orders
  bank_details            == supplier.bank_details_on_file
  invoice_date            <= today, not older than policy allows
  total                   within usual range for supplier
on_fail:
  route:  review
  attach: failing check, page crop, original file

Each check is trivial on its own; together they turn a guess into something a finance team will accept. The failing check travels with the document, so the reviewer knows where to look, and our piece on escalation paths for AI agents covers routing that kind of doubt to the right person.

Confidence you can act on

A single confidence score for a whole document is not actionable: a reviewer cannot tell which fields to distrust, and one threshold sends either too much to review or too little. The useful version is per field, and it comes from agreement between independent signals.

Why not trust the model's own confidence score?

The model's own confidence score is not a dependable measure of whether a value is correct. A language model asked to rate its certainty produces that number like any other text, and it is not reliably calibrated against real accuracy. Some providers expose token probabilities, which carry more information, but they describe how likely the model found the characters it wrote, not whether it read a blurred digit correctly.

Where agreement comes from

Agreement is evidence you can inspect. For each field, collect the signals that support the value independently:

• Two reads agree. Two extraction passes that never saw each other's output give the same value.
• The document agrees with itself. The total in the summary matches the one on the payment slip.
• The checks pass. The amounts satisfy the arithmetic, or the purchase order exists.
• Your records agree. Supplier, bank details and currency match what is on file.

A field with several supporting signals can be accepted automatically, while a field with none, or with conflicting signals, goes to review. Second reads add cost, so spend them on fields that move money or trigger actions, such as totals and bank details, and on any field whose checks failed.

Line items, handwriting and stamps

Header fields such as supplier, date and total are the easy part. Line items, handwritten corrections and stamps are where a mature pipeline still makes mistakes.

How do you extract tables that run across pages?

You extract a table that runs across pages by reading each page's rows separately, recording page and order, and stitching them into one table before any checks run. The stitching rules are specific:

• Drop header rows repeated at the top of each continuation page.
• Remove carried forward and brought forward lines, which would count the same amounts twice.
• Rejoin a row whose description wraps across the page break.
• Keep each row's page and position so a reviewer can find it.

Then run the arithmetic check on the whole stitched table. If the rows sum to the final subtotal, the stitching worked; if not, a gap equal to one row's amount suggests a row was dropped or counted twice. In projects like these, very long tables are where rows tend to go missing when a model reads everything in one pass, so page-by-page extraction with plain-code stitching is usually safer.

Handwritten corrections and stamps

A quantity corrected in pen over a printed one is usually the true value, and exactly the one an extraction model tends to miss, because the printed figure is clearer and looks like everything else on the page. Detect handwritten marks, strikethroughs and initials near extracted values, and send those fields to review with both readings shown. Whether a correction is authoritative is a business rule rather than a reading problem: some teams accept any correction, others only initialled ones.

Stamps work the other way round. They obscure printed values but carry information of their own, such as a received date, a paid mark or an approver's initials, so extract them as fields. A document stamped as paid that arrives for payment deserves a person's attention.

Designing human review

Human review is a permanent part of any extraction system that handles real intake, and its design decides both the running cost and whether the team trusts the output. The principle is to show the evidence and let the reviewer decide quickly. Keep the coordinates of the region every value came from, because a crop of the page beside the extracted value turns a hunt through a scan into a glance.

A good review screen:

• Shows only the doubtful fields, each beside its crop, with the full page one click away.
• Says why each field is there, such as totals disagree or supplier not recognised.
• Pre-fills the best candidate, so accepting takes one keystroke.
• Works from the keyboard, moving to the next field without the mouse.
• Records every decision, accepts included, as labelled data for the evaluation set.

Watch for rubber-stamping: when most pre-filled values are right, reviewers learn to accept without looking. Avoid showing the model's self-reported confidence as a number, and regularly give a sample of automatically accepted documents a full manual check, which is the most direct way to see how many errors get through unnoticed. Route by reason too, so changed bank details reach someone who can phone the supplier.

How should you measure extraction accuracy?

Measure extraction accuracy per field and per source, on a hand-labelled set of your own real documents, and track how much work still reaches a person. A single overall figure averages clean sources with difficult ones and trivial fields with fields that move money, which hides the problems that matter.

Start with an evaluation set: real documents from every intake channel, worst ones included, with the correct value for every field recorded by hand. Keep it separate from anything used to design prompts or rules, and grow it from review decisions. Our guide to testing non-deterministic systems covers building sets like this.

Compare normalised values, so that 1,200.00 and 1200 count as the same total, or formatting noise will drag the figures down. Read the measures together: the straight-through rate can always be raised by accepting more automatically, and only the silent error rate tells you whether that was safe.

What happens when a supplier changes their invoice layout?

When a supplier changes their invoice layout, accuracy drops for that supplier only and the overall figure barely moves. It is a common reason a pipeline works for months and then stops, and it hides in the aggregate because one supplier is a small slice of the total.

Track accuracy and check failures per source every week, and flag known suppliers whose header, totals or table have moved. Plan for the change rather than trying to prevent it: a pipeline that notices quickly, routes that supplier to review while it adapts and records what changed is worth more than one that was slightly more accurate on launch day. The per-supplier view also shows which relationships create the most manual work, often not the ones people assume.

Audit trail and original documents

Every extracted value should be traceable to the document it came from and to every decision made about it. Financial data gets audited, disputed and corrected, and a pipeline that cannot show its working turns a routine query into an investigation. For each document, store:

• The original file as received, with a fingerprint showing it has not changed since intake.
• Each processed version and the steps that produced it.
• Extracted values, printed and normalised, with the page region each came from.
• The versions of prompts, rules and models used.
• Every check that ran, and its result.
• Every human decision: who made it, what changed and when.
• When and where the data went downstream.

Originals also make improvement cheaper, because better preprocessing or extraction can be rerun against stored files without asking suppliers to resend anything. Rerun only the affected sources, since pushing a whole archive back through a language model is rarely worth the cost.

Retention periods for financial records are set by law and vary by country, so agree them with whoever owns compliance. Invoices can contain personal data and bank details, which affects who sees the review queue, where files are stored and which model providers may process them.

Rolling out extraction in stages

Roll out extraction the way you would hand work to a new colleague: check everything at first, then step back as the results earn it. Each stage below produces evidence for the next.

1. Collect real intake. Sample every channel documents arrive through, including the photographs and bundles people apologise for.
2. Agree the schema. Define each field, its format and its absence rule with the people who use the data.
3. Build the evaluation set. Label the sample by hand and split it by source.
4. Test preprocessing alone. Confirm a person can read the processed images before any model is involved.
5. Add extraction and validation. Measure against the evaluation set and fix the weakest fields and sources first.
6. Run in shadow mode. Process live documents beside the existing manual process without posting anything, and compare.
7. Go live with full review. Review every document at first, building labelled data and the team's trust.
8. Automate source by source. Let documents that pass every check skip review where measured accuracy justifies it, and keep sampling.

Where the extracted data lands matters as much as how it is read. Often the destination is still a spreadsheet reconciled by hand, and we cover replacing that in turning a spreadsheet process into a service.

Key takeaways

• Demo accuracy comes from clean files, so test document extraction on real intake, including the worst of it.
• Much of the recoverable accuracy is won in preprocessing, before any model reads the page.
• Classic OCR tends to fail visibly while language models can fail plausibly, which is why many pipelines use both.
• Define every field precisely before writing prompts, and use structured output where the provider supports it.
• Validate with arithmetic, known records and plausibility, and send failures to review with the failing check attached.
• Build per-field confidence from agreement between independent signals rather than the model's own score.
• Measure accuracy per field and per source, alongside the share of documents that need no human touch.
• Keep every original and a full record of decisions, so any value can be traced and reprocessed.

Where to start

Start by looking at the documents. Pull a recent sample from every channel, sort it by source and condition, and note how much of it a person could read at a glance. That shows where preprocessing will matter, which sources will need review and whether the reference data for validation is usable.

If you want help designing or building the pipeline, our document intelligence work covers systems like the one described here. The quickest way to see how your own documents would fare is to schedule a call and bring a few of the worst ones.

Frequently asked questions

Can AI extract data from handwritten documents?

Yes, but handwriting needs more checking than printed text. Vision-capable language models often read clear handwriting well, and some OCR engines include handwriting recognition, though quality varies widely by engine and by writer. The risk is a confident guess at an illegible word or digit. For handwritten values that matter, such as amounts or quantities, compare two independent reads and send disagreements to a person with the crop of the page shown beside the value.

What is the difference between OCR and intelligent document processing?

OCR turns an image of text into characters, while intelligent document processing turns a document into checked, structured data. OCR is one component of it. Intelligent document processing adds the steps around it: preparing the image, classifying and splitting documents, extracting named fields, validating them against arithmetic and business records, routing doubtful values to people and passing results to other systems. Many current pipelines also use language models for the field extraction step.

How accurate is AI invoice data extraction?

It depends on your documents far more than on any published figure, so measure it on your own intake. Clean digital PDFs from regular suppliers are usually handled well, while phone photographs, faxes and handwritten corrections are much harder. Build a hand-labelled sample from every source, then track accuracy per field and per supplier, the share of invoices that need no human touch, and how many errors get through without review.

Do I need a template for every supplier's invoice?

No, although templates still have a place. Language models can find fields on layouts they have not seen before, which removes the need to build a template for each new supplier. For a few high-volume suppliers whose layout never changes, fixed templates applied to OCR output remain cheap and reliable. The trade-off is maintenance, because every template breaks when that supplier redesigns their invoice and someone has to notice and fix it.

Will document extraction work on photos taken with a phone?

Yes, provided the images are corrected before extraction. Phone photos need perspective correction to flatten the page, deskewing based on the text lines, orientation detection and contrast adjusted region by region to deal with shadows. Blur and very low resolution cannot be fully repaired afterwards, so the practical fix is to check image quality at the point of upload and ask the sender for a clearer photo before the document enters the pipeline.

How many documents do I need to test an extraction system?

Enough to cover every source and every condition you receive, rather than a fixed number. A test set drawn only from tidy suppliers will overstate accuracy. Include several examples from each major supplier or channel, plus the hard cases: photographs, bundled files, stamps, handwriting and tables that run across pages. Label the correct values by hand, keep the set separate from anything used for tuning, and keep adding real review decisions to it.

What happens to documents the system cannot read?

They go to a person with the reason attached, rather than being guessed at or silently dropped. A well-designed pipeline marks exactly which fields failed and why, such as an unreadable total or an unrecognised supplier, and shows the reviewer the relevant crop of the page. Documents that cannot be read at all, such as a badly blurred photograph, should trigger a request to the sender for a better copy.

Is it safe to send invoices to an AI model provider?

It can be, but it depends on the provider's terms and your own legal obligations. Check whether the provider keeps submitted data, whether it can be used for training, and where it is processed and stored. Invoices often contain personal data and bank details, so involve whoever owns data protection before going live. Where the terms do not fit, a pipeline can keep OCR and validation in-house and limit what reaches an external model.

Schedule a call