Post-training quantization errors cancel out to protect language model output
Why Does Post-Training Quantization Work?
Machine LearningComputation and Language
Summary
Compressing large language models by lowering the precision of their weights can introduce errors that might build up and mess up the model’s predictions. The authors explain that in pretrained models, errors introduced by one layer often cancel out errors coming from earlier layers, so mistakes don’t pile up as much as expected. Also, the model naturally preserves the most confident predictions despite these small errors. These two effects help explain why post-training quantization works well even though the model wasn’t trained for it.
What this means in practice
- •For machine learning engineers: Compress language models effectively while maintaining prediction accuracy by understanding error cancellation mechanisms in quantization.
- •For mobile app developers: Deploy smaller, efficient language models on devices by leveraging error-resilient post-training quantization for reliable natural language outputs.
Authors
Yuxiang Chen, Michael Beyer, Jun Zhu, Jianfei Chen
Abstract
Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.