Study: LLM compression quietly hurts common knowledge and calibration

A systematic evaluation of 3 LLMs and 11 compression methods, measuring effects on knowledge retention, model confidence, and social bias that aggregate metrics (perplexity, accuracy) mask. Findings: compression disproportionately reduces relative retention of head knowledge compared to tail knowledge; compressed models often remain substantially confident in incorrect answers on newly lost knowledge, so calibration quietly degrades; and stable aggregate bias scores can conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. The conclusion: compression produces asymmetric behavioral changes that aggregate measures cannot see, so granular evaluation is required before deployment.

This is the second paper in two days showing aggregate metrics mask compression damage, after the bitsandbytes proactive-interference result (ev-20260820-02, 8/20). This one is broader — 11 compression methods across 3 models — and adds the confidence/calibration and bias-subgroup dimensions. Together they make granular behavioral probes before deploying compressed models a concrete acceptance-testing requirement rather than just good practice. Confidence-while-wrong is directly dangerous for agent pipelines that branch on model self-assessment.