SWE-Flux contains 480 repository-level questions grounded in instrumented test executions from 12 real Python repositories. Gold answers are harvested from runtime traces instead of written by an LLM judge. The questions cover control flow, loops, state, dataflow, exceptions and invariants across single and multiple tests. Five evaluated LLMs remained weak: the best reached 37% accuracy, and models struggled most with inter-procedural dataflow, precise state and suite-level aggregation.
SWE-Flux tests whether coding models understand runtime behavior
2026-09-24Frontier watchTRIAL
SWE-Flux contains 480 repository-level questions grounded in instrumented test executions from 12 real Python repositories. Gold answers are harvested from runtime traces instead of written by an LLM judge. The questions cover control flow, loops, state, dataflow, exceptions and invariants across single and multiple tests. Five evaluated LLMs remained weak: the best reached 37% accuracy, and models struggled most with inter-procedural dataflow, precise state and suite-level aggregation.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For coding-agent teams, passing tests does not show that a model understands why the program behaved that way. Execution-grounded questions can expose brittle patches and weak debugging reasoning before those failures appear in production. Trial a small internal version using instrumented tests from your own repositories, and refresh variants so the model cannot memorize static answers.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Dataset | 480 instances across 12 real Python repositories |
|---|---|
| Ground Truth | automatically harvested from instrumented test executions |
| Coverage | control flow, loops, state, dataflow, exceptions and invariants; single-test and multi-test |
| Models | five LLMs |
| Best Accuracy | 37% |
| Fresh Variants | valid perturbed variants harvested for almost 90% of selected instances |
Technical impact 3
Engineering value 4
Adoption signal 1
Maturity 2
Verification cost 3
Risk 1