Researchers audit 254 public SWE-bench submissions across four splits using their per-instance verdicts. On Verified, none of the 29 adjacent pairs in the top 30 is distinguishable by an exact paired test. The top ten share 285 successes and 51 failures, leaving only 164 informative tasks. Holding the model fixed, observed scaffold ranges reach 29.8 points, larger than the 8.8-point spread across the top 30.
SWE-bench audit finds top coding-agent ranks statistically unresolved
2026-09-16Frontier watchTRIAL
Researchers audit 254 public SWE-bench submissions across four splits using their per-instance verdicts. On Verified, none of the 29 adjacent pairs in the top 30 is distinguishable by an exact paired test. The top ten share 285 successes and 51 failures, leaving only 164 informative tasks. Holding the model fixed, observed scaffold ranges reach 29.8 points, larger than the 8.8-point spread across the top 30.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams selecting a coding agent, small leaderboard gaps do not justify a procurement decision. Compare model-and-scaffold pairs on an internal workload, report paired uncertainty, and use tiers instead of a strict rank. The released audit pipeline makes that check reproducible without rerunning models.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Dataset | 254 public submissions across Verified, Lite, Test and Multimodal |
|---|---|
| Verified Top 10 | 285 shared successes; 51 shared failures; 164 discriminating instances |
| Paired Test | 0 of 29 adjacent top-30 Verified pairs separable at alpha 0.05 |
| Scaffold Effect | within-model observed range up to 29.8 percentage points |
| Artifact | frozen inputs, scripts, normalized design and tier membership released |
| Limitation | observational audit; non-rejection does not prove equivalence |
Technical impact 4
Engineering value 5
Adoption signal 3
Maturity 3
Verification cost 2
Risk 1