Document AI
Why your invoice OCR works in testing and fails in production
The test folder is clean exports. The real intake is phone photographs. What changes between the two, and how to find out before a client does.
5 min readByteWeave Studio
Every document extraction project has the same first week. Somebody assembles a folder of sample invoices, the pipeline reads them at 96%, and the room relaxes. Then it goes live, accuracy settles somewhere in the seventies, nobody can say exactly why, and within a month the finance team has quietly gone back to typing.
The gap is rarely the model. It is that the sample folder and the real intake are different populations, and the folder was assembled by a person who — without meaning to — picked documents that were easy to read.
Your test set was curated, whether you meant to or not
Ask where the samples came from and the answer is usually that someone exported a few dozen from the accounting system, or forwarded the ones already sitting in their inbox. Both are biased in the same direction. Documents that made it into the system cleanly are over-represented, because the messy ones were the ones that got stuck.
You can see it in the file types. A curated folder skews heavily towards born-digital PDFs with a proper text layer. Real intake in most Indian back offices is a mix: scans from an office multifunction device, phone photographs taken on a warehouse floor, screenshots of a payment confirmation, and occasionally a photograph of a screen displaying a PDF.
What production actually sends you
The failure modes that matter almost never appear in a demo. Pages arrive rotated by ninety degrees or skewed by four. Contrast is poor enough that a decimal point disappears. A stapled invoice is photographed with the second page at an angle, so the line-item table runs off the frame. Someone forwards a scan of a scan, two generations deep.
None of these need a better model. They need preprocessing, and they need the pipeline to notice when a page is unreadable rather than confidently returning a value from it. On our own extraction work, more of the early accuracy gain came from deskewing and contrast normalisation than from anything we did to the extraction step.
Measure the shape of the intake, not just the accuracy
Before quoting an accuracy figure, spend a day characterising what actually arrives. Count the distribution of source types, page counts, resolutions and languages. Count how many documents have no text layer. Count how many are photographs rather than scans. That profile tells you more about what the system will do in month three than any benchmark number.
It is also the honest basis for a conversation with the client. "97% on born-digital PDFs, 84% on phone photographs, and photographs are 40% of your intake" is a useful sentence. A single blended number is not, because it hides which half of the problem you have actually solved.
Build the failure path before you tune the model
A system that is right 97% of the time and silent about the other 3% is worse than one that is right 90% of the time and says which 10% to check. Per-field confidence, a review queue, and a way to see the region of the page a value came from are not polish. They are what makes any accuracy figure usable.
This also changes what you optimise. Once there is a review path, the goal stops being raw accuracy and becomes the share of fields that clear without a human, at an error rate the client can live with. Those are different targets, and the second one is the one that shows up in their operating costs.
A test set that survives contact
Replace the curated folder with a random sample of real intake, drawn across at least a full month so seasonal formats are included. Key it by hand once. It is tedious and it is the highest-value day of work in the project.
Then keep it current. Vendors change their templates, someone switches scanning hardware, a new business unit joins with different paperwork. A ground-truth set assembled in month one and never touched is a slowly rotting measure of a system that keeps changing underneath it.
- OCR
- Document AI
- Accuracy
- Production
More reading
Document AI
Field-level or document-level? How to measure extraction accuracy
The same pipeline can be 97% accurate or 68% accurate depending on how you count. Which denominator to use, and when each one lies.
5 min read
Document AI
Parsing Indian invoices: GSTIN, HSN and the formats that break extraction
Extraction tools trained on US and European paperwork miss the fields that matter here. What is different, and what to validate.
6 min read
LLM engineering
Why your RAG demo works and your RAG product doesn’t
A retrieval demo on twenty documents proves almost nothing about the same system on twenty thousand. The four things that break in between.
6 min read
Have a problem
worth solving?
Tell us what you're building. We'll help you figure out what's possible — and say so if we're not the right people for it.