PASTABench evaluates proactive safety monitoring on 1,139 multi-turn agent trajectories across five risk categories and 13 subcategories. It labels the earliest warning and the actual trigger so monitors are scored on whether, when and why they intervene. Across 16 LLMs, the best system intervened inside the optimal window only 40.74% of the time. Smaller models often lost apparent safety performance when hazard vocabulary was neutralized, exposing keyword matching rather than risk understanding.
PASTABench finds agent safety monitors intervene too late
2026-09-23Frontier watchTRIAL
PASTABench evaluates proactive safety monitoring on 1,139 multi-turn agent trajectories across five risk categories and 13 subcategories. It labels the earliest warning and the actual trigger so monitors are scored on whether, when and why they intervene. Across 16 LLMs, the best system intervened inside the optimal window only 40.74% of the time. Smaller models often lost apparent safety performance when hazard vocabulary was neutralized, exposing keyword matching rather than risk understanding.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams deploying multi-step agents, a post-hoc incident label is not a control. Safety monitors need to catch the trajectory after enough evidence appears but before the harmful transition. Add timing and paraphrase robustness to monitor acceptance tests; do not accept a high aggregate classifier score without checking the intervention window.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Dataset | 1,139 multi-turn trajectories; five risk categories; 13 subcategories |
|---|---|
| Labels | Earliest-Signal turn and Trigger turn |
| Metric | Optimal Intervention Window |
| Models | 16 LLMs |
| Best Result | 40.74% optimal-timing interventions |
| Failure Mode | lexical overfitting exposed by neutralizing hazard vocabulary |
Technical impact 3
Engineering value 4
Adoption signal 1
Maturity 2
Verification cost 3
Risk 2