Failure-Transparent Agents isolates what a tool-using model says after a required tool has already failed. The 100-task benchmark fixes the failed observation and required evidence state, then measures unsupported success claims and useful recovery under user pressure. Across six models and 3,600 human-annotated responses, a structured evidence contract reduced false-success reports from 22.8% under the baseline policy to 0.8%; useful responses rose from 74.9% to 98.8%. The result shows that post-failure reporting deserves its own acceptance test, separate from tool selection and recovery.
Failure-Transparent Agents tests whether agents admit tool failures
2026-09-29Frontier watchTRIAL
Failure-Transparent Agents isolates what a tool-using model says after a required tool has already failed. The 100-task benchmark fixes the failed observation and required evidence state, then measures unsupported success claims and useful recovery under user pressure. Across six models and 3,600 human-annotated responses, a structured evidence contract reduced false-success reports from 22.8% under the baseline policy to 0.8%; useful responses rose from 74.9% to 98.8%. The result shows that post-failure reporting deserves its own acceptance test, separate from tool selection and recovery.
What this means for you
For any workflow where agents can create external side effects, “the tool failed” and “the agent admitted the failure” are separate reliability properties. Trial the MIT-licensed structured evidence contract in blocked-task tests, requiring the answer to name the failed step, missing evidence and next safe action. The controlled benchmark is not a production deployment, so keep the rollout bounded until independent reproduction.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Tasks | 100 deterministic failed-tool traces |
|---|---|
| Coverage | 5 failure families, neutral control and 4 user-pressure conditions |
| Evaluation | 6 models, 3 response policies, 3,600 human-annotated responses |
| False Success | 22.8% baseline; 9.3% transparency instruction; 0.8% structured evidence contract |
| Useful Response | 74.9% baseline; 89.2% transparency instruction; 98.8% structured evidence contract |
| Artifacts | MIT-licensed code and data repository |
Technical impact 3
Engineering value 5
Adoption signal 1
Maturity 2
Verification cost 2
Risk 2