Monitor from outside the work being monitored
An agent should record starts, progress, and terminal outcomes. A separate monitor can compare those records with expected behavior, including runs that stopped before they could report a failure.
A useful watchdog turns findings into deduplicated issues with supporting evidence. Event-driven checks can handle completed runs while a scheduled sweep checks for missing or late outcomes. Detection time depends on the event delivery and sweep intervals configured for that deployment.
That backstop exists for one specific class, and it is the one worth designing around first.
A missing completion event is a finding
A run can die or hang before writing a terminal event. Process supervisors may still detect the crash, but an application monitor that only reads completed-run reports can miss it.
Use deadlines, heartbeats, process health, or a sweep of unfinished runs to detect missing progress. Distinguish a stalled agent from delayed telemetry, and allow for workflows that intentionally wait for a person.
Choose thresholds from the workflow's expected duration and review the resulting false alarms. The goal is an actionable finding, not an alert for every slow run.
The eleven classes
The following taxonomy provides stable names for monitoring and triage. A deployment can use the classes that its instrumentation supports, with thresholds and escalation rules suited to the workflow.
- dangling_run
- Died or hung. No end event was ever written.
- fatal_error
- The run terminated on an unhandled exception.
- fatal_transient
- A transient failure exhausted or bypassed recovery. Review the cause and retry policy before another attempt.
- api_burn
- Structurally identical API errors in a loop. Money burning silently, usually against a provider that is returning a stable error the caller is not reading.
- error_streak
- Repeated errors across consecutive runs. Any single one might be noise; the streak is signal.
- stage_error
- A stage failed and the pipeline continued anyway — the direct generator of confidently incomplete output.
- empty_output
- Reported success with nothing to show for it.
- partial_status
- Self-reported degraded completion. The agent knew it was impaired and said so; something has to read that.
- schema_violation
- Telemetry does not match the expected schema. Check producer and consumer versions as well as the payload.
- latency_ceiling
- Per-run latency budget exceeded.
- cost_ceiling
- Per-run cost budget exceeded.
Four of these eleven — dangling_run, stage_error, empty_output, partial_status — describe runs that did not report themselves as failed. That ratio is the argument for a taxonomy over an error handler.
Keep baseline monitoring available
Start with rules over the available telemetry. They can keep classifying known patterns during a model-provider outage, provided the telemetry pipeline and monitoring infrastructure remain available.
A second, model-assisted pass can inspect findings and a sample of otherwise clean runs for patterns the rules miss. Bound its data access and cost, and measure whether it adds useful detections.
If the second tier is unavailable, retain the first tier and report degraded coverage. This is a monitoring fallback, not permission to bypass business checks or continue a consequential action whose required verification failed.
Fingerprint the failure, not the run
Findings are deduplicated on a hash of the failure signature rather than on the run identifier. One recurring defect becomes one issue with an occurrence counter, instead of a thousand near-identical notifications.
This is not a cosmetic concern. Alert volume is the mechanism by which monitoring stops working: a channel that produces a thousand notifications for one defect gets muted, and the next real incident arrives in a muted channel.
Each issue carries redacted evidence and the run identifiers needed to reproduce the failure — which is the precondition for anything downstream being able to attempt a fix.
Technical examples explain design choices and their assumptions. Deployment controls and acceptance criteria are agreed for each workflow.