Three tiers, and only one of them knows what the work was for
Quality signal in an agent system comes from three places, and they are not equivalent.
Deterministic checks produce proofs. A schema either validates or it does not; a value either matches the source text or it does not. These are the strongest signal available and also the narrowest — they can only assess what you knew in advance to check.
A model judge produces opinions. It can assess things no rule can describe, and it is wrong in ways that are hard to predict.
Neither knows whether the person who asked was served. A document can pass every schema check, satisfy every judge, and still be the wrong answer to the question the operator was actually asking. That is why human reaction is captured as a third and highest tier, and why it supervises the other two rather than being averaged with them.
Capture has to cost the reviewer nothing
The mechanism is a reviewer approving or rejecting an output in the surface they already use. That is the whole capture step. There is no separate labelling tool, no weekly review session, no request for anyone to do work they were not already doing.
This constraint is not a convenience. Any feedback mechanism that adds a task to a busy person's day has a decay curve, and the decay starts about three weeks after launch. The only feedback loop that survives a year is one built on an action the reviewer was going to take anyway.
Every result the agent posts is indexed, so a reaction hours or days later still resolves to the exact run, its inputs, and the reasoning trace that produced it. Late feedback is still usable feedback.
A rejection has to visibly produce something
A rejection is treated as fact rather than opinion. It emits a deterministic finding, jumps the diagnosis queue carrying the context that a human marked this output unsatisfactory, and opens an escalation path.
The reason to wire it that aggressively is behavioural rather than technical. A rejection that visibly produces nothing teaches people to stop rejecting. Once a reviewer concludes that the reject button is decorative, they start silently correcting outputs by hand instead — and at that moment you lose both the signal and the visibility into how much manual correction is actually happening.
The failure is quiet and it looks like success: rejection rates fall, dashboards improve, and the agent is getting worse.
Rejection rate falling is only good news if you can prove people are still rejecting when they should. That proof is a sampling process, not an assumption.
Approvals calibrate the judge
An approval on an output that the model judge had flagged records a disagreement between the human tier and the model tier. Accumulated, those disagreements are a continuously computed false-positive rate for the judge itself.
This matters because the alternative is hand-tuning the quality bar after every review round, which is unfalsifiable work — you adjust a threshold, the numbers move, and nobody can say whether the judge got better or the bar got lower.
A judge with a measured false-positive rate against human ground truth is a component you can reason about. A judge without one is a second opinion of unknown quality being used to gate the first.
The feedback becomes the corpus
Approved outputs are human-confirmed pairs, and they seed the eval set. Rejected outputs, together with their diagnoses, become the failure corpus.
This is what makes the loop compound rather than merely close. The eval suite an agent ships with is a best guess about what will go wrong. The eval suite it has a year later is a record of what actually did — including the edge cases nobody could have anticipated, because they came from the specific vendors, formats and exceptions of one real business.
That corpus is also the most valuable artifact the whole system produces, and it is worth being explicit about who owns it. It describes your work, it accumulates in your environment, and it should leave with you.
These notes describe how Keelo builds. They are written from production systems running in customer environments, and they identify no customer, no vendor, and no deployment. Where a note gives a number, that number is either arithmetic you can reproduce or a threshold Keelo enforces — never a measured customer outcome.