AutoSaddler (arXiv 2608.23041) treats agent-harness improvement as an offline learning problem: it mines failure signals from mini-batches of execution traces, generates structured patches that treat the harness — prompts, tool configurations, control logic — as code, and selects updates through validation. It improved the base harness by 9.0, 9.6 and 10.0 percentage points on GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 respectively. Ablations attribute the gains to deep debugging over shallow reflection and to generalization-aware selection over trajectory-specific repair. Code is released via the project site.
AutoSaddler learns agent-harness fixes from failure traces
2026-08-26Frontier watchTRIAL
AutoSaddler (arXiv 2608.23041) treats agent-harness improvement as an offline learning problem: it mines failure signals from mini-batches of execution traces, generates structured patches that treat the harness — prompts, tool configurations, control logic — as code, and selects updates through validation. It improved the base harness by 9.0, 9.6 and 10.0 percentage points on GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 respectively. Ablations attribute the gains to deep debugging over shallow reflection and to generalization-aware selection over trajectory-specific repair. Code is released via the project site.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For anyone operating coding agents, the harness is becoming a measurable optimization target on par with model choice — and the failure traces you already accumulate are the training data. The recipe is directly borrowable: diagnose failing runs, patch the harness as code, keep only updates that validate on held-out tasks.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Method | offline loop: failure-trace diagnosis -> structured patch generation (harness-as-code) -> validation-based update selection; mini-batch iteration over execution traces |
|---|---|
| Results | +9.0pp GAIA2, +9.6pp SWE-Bench Pro, +10.0pp Terminal-Bench 2.0 over the base harness |
| Ablations | deep debugging > shallow reflection; generalization-aware selection > trajectory-specific repair |
| Code | released via project website (aka.ms domain) |
| Affiliation | project site on aka.ms (Microsoft domain); paper page lists no affiliations |
| Arxiv | 2608.23041, submitted 2026-08-24 |
Technical impact 4
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 2
Risk 2