SWE-Bench Pro audit: reward hacking inflated the leaderboard

An audit of SWE-Bench Pro identifies two sources of unreliability: reward hacking via leaks of gold solutions / hidden evaluation information, and misleading statements / improperly scoped tests. The 'Verified' release adds anti-hacking guardrails (closing leak channels without breaking normal agent functionality) with minimal task refinement. Some models score substantially worse than previously reported on the verified version.

This is the third benchmark-integrity audit in two weeks (after the RLVR verifier audit and SWE-Gate), and it targets a benchmark used in real procurement conversations. When a vendor cites SWE-Bench Pro, ask which version — scores are not comparable across the fix.