Summary
Automated harness evolution usually scores every candidate modification on a fixed task set, letting aggregate scores hide specific regressions. HarnessLens instead jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and verifies each candidate only on behavior-relevant tasks selected through an attributable-evidence gate. Across three agent harnesses and four benchmarks it improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than propose-and-verify baselines. Code is released on GitHub.
Why it matters
This is the second independent result in a week (after AutoSaddler, 8/26) showing the harness is a measurable optimization target, and the new angle is cost: behavior-aware verification makes the evolution loop affordable when each evaluation is expensive. The attribution gate is the piece worth copying into your own eval-driven prompt and tool-tuning loops.
Technical details
| Method | joint exploration of task space + user-configurable components; trajectory-derived candidates; attributable-evidence gate selects behavior-relevant tasks for selective verification |
|---|---|
| Results | +7.6-13.6% average held-out performance across 3 harnesses x 4 benchmarks, at substantially lower evaluation budget than propose-and-verify baselines |
| Relation | converges with AutoSaddler (arXiv 2608.23041, 2026-08-26) on harness-as-optimization-target; differs on budget-aware verification |
| Code | https://github.com/jhxu5214/HarnessLens |
| Arxiv | 2608.27311, submitted 2026-08-27 |
Tags
agent-harnesscoding-agentoptimizationevaluationresearch