AI summaryⓘ
The authors studied how well large language models (LLMs) keep track of their own uncertainty after being made smaller and faster through compression methods like quantization and pruning. They tested 12 different LLMs on five language tasks and measured uncertainty using a special method called conformal prediction. They found that making models smaller often breaks the link between accuracy and uncertainty, bigger models handle this problem better, and the increase in uncertainty happens suddenly rather than gradually. The authors suggest that checking only accuracy is not enough when deciding if a compressed model is ready to use, and it's important to also evaluate how well the model knows when it might be wrong.
Large Language ModelsModel CompressionQuantizationPruningUncertainty EstimationConformal PredictionAccuracyDeploymentBenchmarkingThreshold Effect
Authors
Yujia Tong, Yuxi Wang, Yunyang Wan, Tian Zhang, Junhao Dong, Jingling Yuan
Abstract
Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation. However, in safety-critical applications, a model's ability to reliably quantify its own uncertainty is equally important. We ask: does compression preserve this ability? To answer this question, we benchmark 12 LLMs under various compression configurations across five NLP tasks, using conformal prediction to provide a rigorous, distribution-free measure of uncertainty. Our experiments reveal that: (I) compression frequently decouples accuracy from uncertainty; (II) larger models absorb compression-induced uncertainty far more effectively than smaller ones; and (III) uncertainty inflation is often threshold-like rather than gradual. These results suggest that accuracy-only evaluation is insufficient for assessing the deployment readiness of compressed LLMs, and that uncertainty-aware benchmarking should be a standard component of model compression pipelines.