Automated harness evolution usually scores every candidate modification on a fixed task set, letting aggregate scores hide specific regressions. HarnessLens instead jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and verifies each candidate only on behavior-relevant tasks selected through an attributable-evidence gate. Across three agent harnesses and four benchmarks it improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than propose-and-verify baselines. Code is released on GitHub.
HarnessLens evolves agent harnesses under a verification budget
2026-08-29Frontier watchWATCH
Automated harness evolution usually scores every candidate modification on a fixed task set, letting aggregate scores hide specific regressions. HarnessLens instead jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and verifies each candidate only on behavior-relevant tasks selected through an attributable-evidence gate. Across three agent harnesses and four benchmarks it improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than propose-and-verify baselines. Code is released on GitHub.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
This is the second independent result in a week (after AutoSaddler, 8/26) showing the harness is a measurable optimization target, and the new angle is cost: behavior-aware verification makes the evolution loop affordable when each evaluation is expensive. The attribution gate is the piece worth copying into your own eval-driven prompt and tool-tuning loops.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Method | joint exploration of task space + user-configurable components; trajectory-derived candidates; attributable-evidence gate selects behavior-relevant tasks for selective verification |
|---|---|
| Results | +7.6-13.6% average held-out performance across 3 harnesses x 4 benchmarks, at substantially lower evaluation budget than propose-and-verify baselines |
| Relation | converges with AutoSaddler (arXiv 2608.23041, 2026-08-26) on harness-as-optimization-target; differs on budget-aware verification |
| Code | https://github.com/jhxu5214/HarnessLens |
| Arxiv | 2608.27311, submitted 2026-08-27 |
Technical impact 3
Engineering value 4
Adoption signal 1
Maturity 2
Verification cost 2
Risk 2