EVOHARNESSBENCH (arXiv 2609.04280, Mon 7 Sep digest) is a benchmark for evaluating agents under controlled harness evolution — placing non-stationarity not in the task stream (as continual-learning agent benchmarks do) but in the externally supplied harness of tools, skills and agents itself. It contains 17 deterministic multi-stage harness streams built from verifier-based benchmarks: 802 tasks, 520 tools, 42 skills, 62 agents. Two complementary settings isolate the central challenges: deployment evaluation (does previously accessible competence survive as the harness expands?) and self-evolving adaptation evaluation (does accumulated experience transfer to unseen harness configurations?).
EVOHARNESSBENCH measures what agents lose as their harness evolves
2026-09-08Frontier watchTRIAL
EVOHARNESSBENCH (arXiv 2609.04280, Mon 7 Sep digest) is a benchmark for evaluating agents under controlled harness evolution — placing non-stationarity not in the task stream (as continual-learning agent benchmarks do) but in the externally supplied harness of tools, skills and agents itself. It contains 17 deterministic multi-stage harness streams built from verifier-based benchmarks: 802 tasks, 520 tools, 42 skills, 62 agents. Two complementary settings isolate the central challenges: deployment evaluation (does previously accessible competence survive as the harness expands?) and self-evolving adaptation evaluation (does accumulated experience transfer to unseen harness configurations?).
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Every team shipping agent platforms adds tools weekly; this is the first controlled instrument for the resulting regression class — competence quietly lost when a harness grows — plus the other direction: whether experience gained on one harness config transfers after the next platform update. The deterministic stream construction (verifier-based) means it can run in CI as a harness-change canary, which is exactly how AutoSaddler/HarnessLens-style findings should be operationalized.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Construction | 17 deterministic multi-stage harness streams from verifier-based benchmarks; axes: tools, skills, agents |
|---|---|
| Scale | 802 tasks, 520 tools, 42 skills, 62 agents |
| Settings | deployment evaluation (retention of previously accessible competence as the harness expands) + self-evolving adaptation evaluation (transfer of accumulated experience to unseen harness configurations) |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 3
Verification cost 2
Risk 1