A trajectory-aware method selects coding-agent regression cases from embeddings of prior step-by-step runs instead of drawing a random subset. A 10% subset keeps median resolve-rate estimation error below 5% while cutting token use by roughly 90%. Across configuration, model and framework changes, the method also lowers worst-case error by 38% to 46% relative to a bad draw from the strongest random baseline.
Trajectory sampling cuts coding-agent regression cost by about 90%
2026-09-22Frontier watchWATCH
A trajectory-aware method selects coding-agent regression cases from embeddings of prior step-by-step runs instead of drawing a random subset. A 10% subset keeps median resolve-rate estimation error below 5% while cutting token use by roughly 90%. Across configuration, model and framework changes, the method also lowers worst-case error by 38% to 46% relative to a bad draw from the strongest random baseline.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams rerunning SWE-agent benchmarks on every release, full-suite evaluation can consume hundreds of millions of tokens. Trajectory-aware subsets are a practical design for a fast CI gate, but they should be paired with scheduled full runs because the selected cases can age when models and scaffolds change. No public implementation is linked, so plan to reproduce the method before relying on the reported savings.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Method | deterministic selection near outcome-group centroids in trajectory-embedding space |
|---|---|
| Evaluation | 76 subset configurations across same-config, model/config and framework-change regressions |
| Subset | 10% of benchmark cases |
| Median Error | below 5% |
| Token Reduction | about 90% |
| Worst Case Gain | 38%-46% lower error than the 95th-percentile draw of the strongest baseline |
Technical impact 3
Engineering value 5
Adoption signal 1
Maturity 1
Verification cost 4
Risk 2