Expert audit finds physics benchmarks understate frontier models

Domain experts re-grade six physics benchmarks for GPT-5.6 Sol, Fable 5 and Gemini 3.1 Pro, separating model errors from broken questions, reference solutions and graders. On audited public-source cases initially marked wrong, 148 of 152 are benchmark or grader errors. After repair or exclusion, GPT-5.6 Sol rises from 47.3% to 78.7% mean@4 on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark; corrected pass@4 on retained CritPt tasks reaches 94.4%. The corrected scores use retained or repaired subsets, so they are not a direct like-for-like leaderboard replacement.

For teams using scientific benchmarks to route models or claim capability gaps, evaluator and reference quality are now large enough to dominate the measured error. High-stakes benchmark suites need expert adjudication, equivalent-answer handling and retained-set reporting. Near-saturation of repaired closed-ended tasks also means new evaluations must test open-ended modeling and experimental judgment.