The paper reports that OnTrack monitors SWE-bench trajectories at about one millisecond per step and saves about 18% of failed-run compute.
OnTrack flags failing agent trajectories in about one millisecond per step
2026-10-09Frontier watchWATCH
OnTrack continuously compares an agent trajectory with the structure of successful runs to warn or stop failures during execution. On SWE-bench traces, the authors report about one millisecond per step, a 0.057 AUROC gain over content similarity after eight steps, and about 18% less compute spent on failed runs.
Availability: paper only; no public code found
What this means for you
For long-running agent platforms, this suggests a cheaper online guardrail than calling an LLM monitor at every step. No implementation is public and the abort experiment triggered only six times, so it is a watch item until code or larger replication appears.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Latency | about 1 ms per step |
|---|---|
| Evaluation | SWE-bench trajectories |
| Reported Gain | +0.057 AUROC after eight steps |
| Reported Savings | about 18% failed-run compute; 5 of 6 aborts were correct |
| Availability | paper only; no public code found |
Technical impact 3
Engineering value 4
Adoption signal 1
Maturity 1
Verification cost 4
Risk 2