Summary
AutoSaddler (arXiv 2608.23041) treats agent-harness improvement as an offline learning problem: it mines failure signals from mini-batches of execution traces, generates structured patches that treat the harness — prompts, tool configurations, control logic — as code, and selects updates through validation. It improved the base harness by 9.0, 9.6 and 10.0 percentage points on GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 respectively. Ablations attribute the gains to deep debugging over shallow reflection and to generalization-aware selection over trajectory-specific repair. Code is released via the project site.
Why it matters
For anyone operating coding agents, the harness is becoming a measurable optimization target on par with model choice — and the failure traces you already accumulate are the training data. The recipe is directly borrowable: diagnose failing runs, patch the harness as code, keep only updates that validate on held-out tasks.
Technical details
| Method | offline loop: failure-trace diagnosis -> structured patch generation (harness-as-code) -> validation-based update selection; mini-batch iteration over execution traces |
|---|---|
| Results | +9.0pp GAIA2, +9.6pp SWE-Bench Pro, +10.0pp Terminal-Bench 2.0 over the base harness |
| Ablations | deep debugging > shallow reflection; generalization-aware selection > trajectory-specific repair |
| Code | released via project website (aka.ms domain) |
| Affiliation | project site on aka.ms (Microsoft domain); paper page lists no affiliations |
| Arxiv | 2608.23041, submitted 2026-08-24 |
Tags
agent-harnesscoding-agentoptimizationexecution-tracesswe-benchresearch