Document AI
Field-level or document-level? How to measure extraction accuracy
The same pipeline can be 97% accurate or 68% accurate depending on how you count. Which denominator to use, and when each one lies.
5 min readByteWeave Studio
A twenty-field invoice extracted with one wrong field is 95% accurate by field and 0% accurate by document. Both numbers describe the same run. Which one you quote changes the conversation entirely, and quoting the flattering one without saying which it is has soured more client relationships than genuine underperformance.
The right answer depends on what happens downstream, and it is worth deciding before anyone builds a dashboard around it.
Field-level accuracy, and where it flatters
Counting per field is the fairer measure of the model itself. A page with nineteen correct values and one uncertain one is not a failure, it is one value for a person to confirm. It also lets you see which fields are hard, and the answer is usually consistent: totals and dates are easy, party names are middling, and anything hand-written or rubber-stamped is where the errors live.
It flatters when fields are unevenly important. Getting the vendor address right and the invoice total wrong is not 50% success, it is a failed document. If you report field accuracy, weight it or at least report the critical fields separately.
Document-level accuracy, and where it misleads
Counting whole documents as pass or fail matches what an operations team experiences: either this invoice needed a human or it did not. It is the number that maps to cost, so it is usually the one to put in a contract.
It misleads in the other direction. A pipeline improving from three errors per document to one error per document shows no movement at all under a strict all-or-nothing measure, even though the review workload has fallen by two thirds. Teams have abandoned real improvements because the headline metric did not move.
The measure that actually matters
For most production systems the useful metric is neither: it is straight-through rate at a bounded error rate. What share of documents completed with no human involvement, and of those, how many were wrong? That pairs the thing the client is buying with the risk they are accepting.
It also makes the confidence threshold visible as the lever it is. Straight-through rate and error rate move against each other, and the right operating point is a business decision about what a mistake costs, not a technical one.
Getting the denominator honest
Decide up front how to count a field that is genuinely absent from the document, because different rules produce very different totals. A missing purchase-order number correctly returned as empty should count as correct; counting it as a miss punishes the system for reading the page properly.
Do the same for unreadable documents. If a page is too degraded for a human to key, it should sit in its own bucket rather than in the error rate. Agree this with the client while everyone is relaxed, not during acceptance testing.
- Document AI
- Evaluation
- Metrics
- OCR
More reading
Document AI
Why your invoice OCR works in testing and fails in production
The test folder is clean exports. The real intake is phone photographs. What changes between the two, and how to find out before a client does.
5 min read
Document AI
Parsing Indian invoices: GSTIN, HSN and the formats that break extraction
Extraction tools trained on US and European paperwork miss the fields that matter here. What is different, and what to validate.
6 min read
LLM engineering
Why your RAG demo works and your RAG product doesn’t
A retrieval demo on twenty documents proves almost nothing about the same system on twenty thousand. The four things that break in between.
6 min read
Have a problem
worth solving?
Tell us what you're building. We'll help you figure out what's possible — and say so if we're not the right people for it.