EarlyEval (arXiv 2609.02783, Thu 3 Sep digest; SMU) reduces the cost of agent evaluation by predicting outcomes early: LightGBM success/failure classifiers over behavioral, textual, and reference-solution features stop an agent run once confidence crosses a calibrated threshold. On SWE-bench Verified, TerminalBench, and Toolathlon it eliminates 13-26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens at 89-97% accuracy, perturbing per-agent solve rates by only 1-2 points. Code and data are public.
EarlyEval cuts agent evaluation cost by stopping doomed runs early
2026-09-04Frontier watchTRIAL
EarlyEval (arXiv 2609.02783, Thu 3 Sep digest; SMU) reduces the cost of agent evaluation by predicting outcomes early: LightGBM success/failure classifiers over behavioral, textual, and reference-solution features stop an agent run once confidence crosses a calibrated threshold. On SWE-bench Verified, TerminalBench, and Toolathlon it eliminates 13-26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens at 89-97% accuracy, perturbing per-agent solve rates by only 1-2 points. Code and data are public.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Agent evaluation is a real line item: frontier-agent runs cost dollars to hundreds of dollars each, and most of that spend is on runs that were doomed by step three. This is a drop-in saver for any team running agent evals at scale — the 1-2-point solve-rate perturbation is the honest cost of a 30-40% token cut.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2609.02783, announced in the Thu 3 Sep 2026 digest |
|---|---|
| Org | Singapore Management University (Yuling Shi, David Lo) |
| Mechanism | LightGBM success/failure classifiers over behavioral/textual/reference-solution features; calibrated confidence threshold triggers early stop |
| Results | SWE-bench Verified / TerminalBench / Toolathlon: 13-26% agent steps eliminated; up to 44.1% input tokens and 29.4% output tokens saved; 89-97% accuracy; per-agent solve rates perturbed 1-2 points |
| Code | public (per abs-page comments) |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 2
Risk 1