The arithmetic is a product, not an average
A production agent is rarely one model call. It is a pipeline: ingest the document, recover its structure, extract fields, normalize units, resolve entities against a system of record, apply business rules, assemble the output, and write it somewhere. Eight stages is unremarkable — many real workflows run more.
End-to-end accuracy is the product of the stages, not their mean. If each of eight stages is independently 95% accurate, the pipeline returns 0.95⁸ — about 66%. One output in three is wrong somewhere.
The reason this defect is so durable is that every stage passes review. A 95% stage looks healthy on its own dashboard. An engineer who tests each stage in isolation finds nothing wrong, ships, and then watches a third of production output come back rejected. Nobody built a bad stage. The pipeline was arithmetically incapable of the accuracy that was promised for it.
- 8 stages at 95% each
- 66%
- Required per stage for 95% end to end
- 99.4%
- Required per stage for 99% end to end
- 99.9%
99.4% is not a prompting problem
Invert the equation and the engineering agenda falls out of it. For an eight-stage pipeline to clear 95% end to end, each stage has to hold 99.4%. For 99% end to end, each stage needs 99.9%.
No amount of prompt iteration reliably moves a reasoning step from 95% to 99.4%. That is a different regime, and there are only three ways into it.
Remove the stage. Every stage you delete removes a multiplicand. Collapsing two reasoning steps into one, or dropping a step whose output nothing downstream consumes, buys more accuracy than any prompt rewrite will.
Make the stage deterministic. A regex, a schema constraint, a lookup against a closed vocabulary, a tiered string match — these are not 99.4% accurate, they are correct. Every stage moved from a reasoning step to a deterministic one stops being a multiplicand at all.
Verify the stage independently. A stage whose output is re-checked against the source by a separate path has a much higher effective accuracy than the same stage unverified, because the verification catches the residual instead of passing it downstream.
The useful reframing: you are not trying to make a model smarter. You are trying to reduce the number of places where being smart is load-bearing.
Deterministic first, then reason over the residual
The pattern this produces is consistent enough to be a rule. Deterministic passes run first and narrow the space — tiered matching, closed vocabularies, regex extraction, schema constraints. Whatever survives that is handed to the model, which now faces a genuinely ambiguous residual rather than the whole input.
Across the hardest agents built so far, the two biggest engineering sinks both converged the same way: from asking a frontier model to diagnose the whole problem, to deterministic matching that escalates only real conflicts. In both cases the accuracy improvement and the cost reduction arrived together, because they have the same cause.
This is also why per-task model routing matters more than model choice. Architecture and genuinely ambiguous exception handling are judgment problems worth a frontier model. Extraction, classification and normalization are volume problems that should run on a cheaper tier — and, wherever possible, should not run on a model at all.
The model must never grade its own work
An agent asked whether its output is good says yes. Self-assessed confidence is not a measurement; it is the same generative process that produced the error, asked a second time.
Verification has to be a separate path — a deterministic re-check against the source text layer, or an independent model pass — and it has to emit a hard pass or fail rather than a feeling. Every extracted value should be resolvable back to the region of the source it came from.
The corollary is that a confidence score between zero and one is close to useless to the person reviewing the output. It tells a reviewer nothing they can act on. What a reviewer needs is a derivation story: not how sure the system is, but why it believes what it believes.
Confidence is a ladder, not a number
Replacing the scalar with a deterministic ladder makes review tractable. Each tier states how the value was established, and each carries a prescribed action.
- Text-confirmed
- The value was re-read and matched in the source text layer.
- Vocabulary-matched
- It resolved against a closed, curated vocabulary.
- Cross-model agreed
- Two independent passes returned the same value.
- Misread-corrected
- A known deterministic correction was applied.
- Model-only
- Reasoned, with nothing independent confirming it.
- Contradicted
- Sources disagree. Escalated, never guessed.
A reviewer sorting by tier is doing triage. A reviewer sorting by a decimal is guessing where the cutoff should be.
Silent failure is the expensive class
The costly bug is not the one that crashes. It is output that is quietly incomplete and looks finished — a field that came back empty because a parse failed three stages up, sitting in a report that renders perfectly.
Every internal state that matters has to be wired to a surface a person actually sees. A blank field carrying a note about why it is blank beats a confident guess in every case, because the blank field gets caught and the guess gets approved.
Reproducibility belongs in the same category. Temperature is zero on every call in a validation path. Two runs over the same input return the same answer, or a reviewer stops trusting the output — and an agent nobody trusts is an agent nobody uses, whatever its accuracy number says.
The ship gate is a holdout nobody has looked at
None of the above is verifiable without an eval corpus, and the corpus is where the discipline actually lives.
Golden sets guide development and can be inspected freely. A separate holdout decides release, and it is never opened while iterating — the moment you look at it, it stops measuring generalization and starts measuring memorization. A regression against baseline blocks release regardless of the absolute number.
Building that corpus is consistently the longest pole in the work. Longer than the scorer, longer than the prompt, longer than the pipeline it grades. Teams that treat evals as something to add after the first production failure have usually already shipped the pipeline that returns 66%.
These notes describe how Keelo builds. They are written from production systems running in customer environments, and they identify no customer, no vendor, and no deployment. Where a note gives a number, that number is either arithmetic you can reproduce or a threshold Keelo enforces — never a measured customer outcome.