SWE-Gate: a third of functionally passing agent fixes fail review constraints

SWE-Gate (arXiv 2609.04167, Fri 4 Sep digest, cs.AI) is a benchmark of 303 repository-level fix instances across 75 open-source Python repositories, each carrying review constraints derived from real PR review comments, with functional correctness and constraint compliance separated out — independent functional and constraint tests, plus non-compliant and golden patches. On a common scaffold across four LLM backbones, of 644 fixes that passed the functional tests, 221 failed the review constraints. A replication package is public.

'The tests pass' is the current acceptance bar for coding agents, and this quantifies exactly how far it sits from 'a human would accept this patch' — one in three. Teams shipping agent-generated code should add constraint-style checks (derived from their own review conventions) to their eval harnesses; the benchmark and its non-compliant/golden-patch pairs are directly usable.