The agent cannot be the thing that judges the agent
Every agent emits per-run structured telemetry on exit. A separate watchdog process reads it. That separation is the entire design: code that is failing is not in a position to assess whether it is failing, and an agent that crashed mid-run will not be filing a report about it.
The watchdog turns a failure into a filed, deduplicated, evidence-carrying issue — ideally before anyone downstream notices the output was wrong. Its detection loop picks up each run manifest within seconds of the run closing, with a slower sweep behind it as a backstop.
That backstop exists for one specific class, and it is the one worth designing around first.
The most important failure class raises no error
A run that dies or hangs without ever writing an end event produces no exception, no stack trace, and no alert. Nothing failed loudly. There is simply a run that started and no record that it finished.
Exception-based monitoring is structurally blind to it. The only way to catch it is to treat the absence of an expected terminal event as itself an event — which requires a sweep that reasons about runs that should have closed and did not, rather than a handler that reacts to errors that were raised.
In practice this class is over-represented in the incidents that reach a customer, precisely because it is the one that alerting misses.
The eleven classes
Named classes beat a generic health check because a reader can check them against their own incident history, and because a fix loop can dispatch on them. The taxonomy below is the one Keelo's watchdog classifies against.
- dangling_run
- Died or hung. No end event was ever written.
- fatal_error
- The run terminated on an unhandled exception.
- fatal_transient
- Killed by something the retry policy should have absorbed — a retry policy that is not doing its job is a defect, not weather.
- api_burn
- Structurally identical API errors in a loop. Money burning silently, usually against a provider that is returning a stable error the caller is not reading.
- error_streak
- Repeated errors across consecutive runs. Any single one might be noise; the streak is signal.
- stage_error
- A stage failed and the pipeline continued anyway — the direct generator of confidently incomplete output.
- empty_output
- Reported success with nothing to show for it.
- partial_status
- Self-reported degraded completion. The agent knew it was impaired and said so; something has to read that.
- schema_violation
- The instrumentation itself broke. Telemetry that no longer conforms is telemetry that is no longer being read correctly.
- latency_ceiling
- Per-run latency budget exceeded.
- cost_ceiling
- Per-run cost budget exceeded.
Four of these eleven — dangling_run, stage_error, empty_output, partial_status — describe runs that did not report themselves as failed. That ratio is the argument for a taxonomy over an error handler.
Deterministic tier first, semantic tier second, and it must fail open
Classification runs in two tiers. The first is rule-based against the taxonomy: no model, no token cost, no dependency on an external service being up. This tier alone catches the majority of what goes wrong, and it keeps working during exactly the incidents that would take a model-based monitor offline.
The second tier is a model pass over the findings plus a small sample of clean runs, to catch what rules cannot describe. It runs under a hard daily spend cap.
The ordering constraint matters more than either tier: the semantic tier must fail open to the deterministic one. A budget-exhausted or unavailable Tier 2 should degrade the watchdog, never blind it. A monitoring system with a soft dependency on a paid API has an outage mode where it stops reporting and looks healthy.
Fingerprint the failure, not the run
Findings are deduplicated on a hash of the failure signature rather than on the run identifier. One recurring defect becomes one issue with an occurrence counter, instead of a thousand near-identical notifications.
This is not a cosmetic concern. Alert volume is the mechanism by which monitoring stops working: a channel that produces a thousand notifications for one defect gets muted, and the next real incident arrives in a muted channel.
Each issue carries redacted evidence and the run identifiers needed to reproduce the failure — which is the precondition for anything downstream being able to attempt a fix.
These notes describe how Keelo builds. They are written from production systems running in customer environments, and they identify no customer, no vendor, and no deployment. Where a note gives a number, that number is either arithmetic you can reproduce or a threshold Keelo enforces — never a measured customer outcome.