SWE-Gate (arXiv 2609.04167, Fri 4 Sep digest, cs.AI) is a benchmark of 303 repository-level fix instances across 75 open-source Python repositories, each carrying review constraints derived from real PR review comments, with functional correctness and constraint compliance separated out — independent functional and constraint tests, plus non-compliant and golden patches. On a common scaffold across four LLM backbones, of 644 fixes that passed the functional tests, 221 failed the review constraints. A replication package is public.
SWE-Gate: a third of functionally passing agent fixes fail review constraints
2026-09-05Frontier watchTRIAL
SWE-Gate (arXiv 2609.04167, Fri 4 Sep digest, cs.AI) is a benchmark of 303 repository-level fix instances across 75 open-source Python repositories, each carrying review constraints derived from real PR review comments, with functional correctness and constraint compliance separated out — independent functional and constraint tests, plus non-compliant and golden patches. On a common scaffold across four LLM backbones, of 644 fixes that passed the functional tests, 221 failed the review constraints. A replication package is public.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
'The tests pass' is the current acceptance bar for coding agents, and this quantifies exactly how far it sits from 'a human would accept this patch' — one in three. Teams shipping agent-generated code should add constraint-style checks (derived from their own review conventions) to their eval harnesses; the benchmark and its non-compliant/golden-patch pairs are directly usable.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2609.04167, announced in the Fri 4 Sep 2026 digest (cs.AI pool) |
|---|---|
| Benchmark | 303 repository-level fix instances, 75 open-source Python repos; review constraints derived from real PR review comments |
| Design | functional correctness and constraint compliance measured independently; non-compliant and golden patches included |
| Result | common scaffold, 4 LLM backbones: of 644 functionally passing fixes, 221 failed review constraints (~34%) |
| Code | replication package public under github.com/DeepSoftwareAnalytics |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 2
Risk 1