Aggregate benchmark gains hide per-item regressions in LLM API migrations

An empirical software-engineering study of what aggregate benchmark scores hide during commercial LLM API migrations. The authors ran three pairwise upgrades in the GPT-5.4 → GPT-5.6 Sol product sequence over 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following), querying each item 50 times per model, and classified every item as reliably improved, reliably regressed, practically equivalent, or inconclusive under false-discovery-rate control with a practical-significance threshold. Reliable improvements and regressions coexist in all nine migration-benchmark cells: upgrade edges with aggregate gains of up to 7.3 percentage points still contain up to 8.3% reliably regressed items, and aggregate-loss edges contain up to 10.7% reliably improved ones. On instruction following, the latest migration's 3.9-point regression under strict scoring shrinks to 0.04 under loose scoring, so scoring strictness alone can make a regression disappear. The complete response-level archive and per-item scoring outputs are released.

Every team on a commercial LLM API faces forced migrations when vendors deprecate models. This paper turns 'aggregate scores look fine' into a measured, reproducible warning: net-positive upgrades still ship double-digit percentages of reliably regressed behaviors, and scoring strictness alone can hide them. For API consumers, the methodology — per-item repeated sampling, FDR control, practical-significance thresholds, plus a released archive — is directly reusable as a migration acceptance gate, which is more useful than another leaderboard.