'Where the Verifier Fails' (arXiv 2609.01354, Wed 2 Sep digest, sole author Esther Xin) audits the automatic verifiers that RLVR and benchmark evaluation depend on, using metamorphic testing: rewrites that preserve mathematical meaning by construction, so any rejection is a provable false negative. Across four widely used verifiers and 307,420 verdicts: self-validation rates range from 53.8% to 95.2% on identical inputs (a 41.3-point spread); two configurations of the same library disagree on 49.9% of pairs; whitespace and punctuation account for 93.0% of in-contract failures under the default LaTeX configuration (trailing periods and newlines dominate); and one numeric cascade accepts off-by-one wrong answers as a step function of magnitude — 0% below 10^4, 100% at or above — due to scale-invariant relative tolerance. Code, the transform suite, the contract matrix, and per-sample verdict records are public.
Verifier audit: RLVR reward errors concentrate in whitespace and punctuation
'Where the Verifier Fails' (arXiv 2609.01354, Wed 2 Sep digest, sole author Esther Xin) audits the automatic verifiers that RLVR and benchmark evaluation depend on, using metamorphic testing: rewrites that preserve mathematical meaning by construction, so any rejection is a provable false negative. Across four widely used verifiers and 307,420 verdicts: self-validation rates range from 53.8% to 95.2% on identical inputs (a 41.3-point spread); two configurations of the same library disagree on 49.9% of pairs; whitespace and punctuation account for 93.0% of in-contract failures under the default LaTeX configuration (trailing periods and newlines dominate); and one numeric cascade accepts off-by-one wrong answers as a step function of magnitude — 0% below 10^4, 100% at or above — due to scale-invariant relative tolerance. Code, the transform suite, the contract matrix, and per-sample verdict records are public.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Every RLVR pipeline and every benchmark leaderboard sits on top of one of these verifiers, and the failure modes are exactly the boring ones: trailing periods, newlines, tolerance regimes. For teams running RLVR, the audit is cheap to replicate on your own reward stack before trusting reward curves — a wrong verifier silently distorts both training signal and eval rankings, and two configs of the same library disagreeing on half of pairs means verifier config is itself an experiment variable.
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2609.01354, announced in the Wed 2 Sep 2026 digest (submitted 9/1) |
|---|---|
| Method | metamorphic testing applied to the verifier: certified-equivalent answer rewrites, so any rejection is a provable false negative |
| Scale | 4 widely used verifiers, 307,420 verdicts |
| Findings | self-validation inconsistency 53.8%-95.2% (41.3pt spread); same-library config disagreement 49.9% of pairs; whitespace+punctuation = 93.0% of in-contract failures (default LaTeX config; trailing periods/newlines dominate); numeric cascade accepts off-by-one wrong answers as a step function of magnitude (0% below 10^4, 100% at/above) from scale-invariant relative tolerance |
| Code | github.com/ethxin0011/verifier-error-budget (code, transform suite, contract matrix, per-sample verdict records) |