Automating Document Processing With AI: A Practical Guide
Invoices, contracts, applications, claims. The architecture that works, the failure modes nobody warns you about, and how to know when it is ready.
Document processing is the most common automation request we get, and the one where expectations and reality diverge most sharply. The demo — upload a clean PDF, watch structured data appear — is genuinely easy now. The production system is not, and the gap is almost entirely in the parts nobody demos.
The pipeline that works
Six stages. Each one a durable checkpoint, so a failure at stage five does not discard stages one through four.
1. Ingestion. Documents arrive from email, a watched folder, an upload, or an API. This stage does one job: get the artefact into durable storage, record its provenance, and emit an event. Never parse here. When something goes wrong downstream, you want the original file exactly as received.
2. Normalisation. Everything becomes a consistent internal representation. Native PDFs get text extracted directly. Scans go through OCR. Images get preprocessed — deskewed, contrast-adjusted. Office formats get converted. The output of this stage is uniform regardless of what came in.
3. Classification. What kind of document is this? Cheap, fast, and enormously valuable, because it lets everything downstream be specialised. A model with a short prompt handles this well, and a small model handles it well enough at a fraction of the cost.
4. Extraction. The actual work: pull the fields you need into a defined schema. Extraction quality depends far more on how you frame the task than on which model you use. Ask for a strict schema. Provide the document with layout information preserved where possible. Require a null rather than a guess when a field is genuinely absent.
5. Validation. Extracted data gets checked against rules that have nothing to do with the model. Do the line items sum to the total? Is the date within a plausible range? Does the reference number match the expected format? Does this supplier exist? Most extraction errors are caught here, cheaply and deterministically.
6. Routing. Confident and valid goes straight through. Anything else goes to a review queue with the specific problem flagged and the source document displayed next to the extracted values.
The failure modes nobody mentions
Documents that are not documents. A “PDF invoice” that is a photo of a screen. A scan at 150 DPI where the decimal points are ambiguous. A fax-to-PDF that has been through three generations of compression. These are common in exactly the industries most eager to automate, and no amount of prompting fixes an unreadable input. Detect and route rather than attempting heroics.
Multi-document files. One PDF containing an invoice, two delivery notes, and a signed page from something unrelated. If you assume one file equals one document, this corrupts your data silently. Splitting is its own classification step.
Tables that span pages. Line items broken across a page boundary, with the header repeated, or not. This is a genuinely hard layout problem and it is where most extraction accuracy is lost on complex documents.
Values that look right and are not. A confidently extracted total that came from the wrong column. This is the dangerous category, because it passes every check you did not think to write. It is the reason validation rules should encode business logic — sums, cross-references, plausible ranges — rather than just checking that a number was found.
Format drift. A supplier changes their invoice template and your accuracy quietly drops for that supplier only, with no error raised anywhere. Per-source accuracy monitoring catches this. Aggregate accuracy monitoring does not.
Confidence and the review queue
The single most important design decision is what happens when the system is unsure, and the answer is never “guess.”
Confidence should be composed from several signals, not taken from the model’s self-report alone. Did validation pass? Are the extracted values internally consistent? Does this document match a known template? How does the extraction compare to previous documents from the same source? Combined, these are far better calibrated than any one of them.
Then route on it. High confidence proceeds automatically. Everything else lands in a review queue — and the quality of that queue determines whether the whole project succeeds. A good one shows the document and the extracted fields side by side, highlights the specific field that failed, allows correction in a keystroke or two, and takes seconds rather than minutes. A bad one is a spreadsheet of rejected items that nobody opens.
The corrections from that queue are also your best source of training and evaluation data. Capture them.
What accuracy to expect
Be sceptical of any number quoted without a document set attached. Accuracy on clean, consistent, native-PDF documents from a handful of known sources is very high. Accuracy on mixed-quality scans from hundreds of sources is meaningfully lower. Both are described as “document AI.”
The honest framing is not a percentage. It is: what proportion of documents can go through without a human touching them, and how long does the remainder take to review? A system that clears seventy percent automatically and makes the other thirty percent take fifteen seconds each is transformative. A system that clears ninety-five percent but makes the remaining five percent take ten minutes of investigation may be worse than what you had.
Rolling it out without risking anything
Run in shadow first. The pipeline processes real documents alongside the existing manual process, and nothing it produces is used. Compare outputs. Every disagreement is either a bug or an ambiguity in your own definition of correct — and you want to find both before cutover, not after.
Then go segment by segment. Start with your highest-volume, most consistent document source, because that is where the accuracy will be best and the value highest. Widen only when the numbers hold.
Keep the manual path available throughout. The confidence to switch off a working process comes from watching the new one match it for a few weeks, not from a go-live date in a plan.
This is a large part of what we build. See our approach to automation, or read about when custom beats no-code.