Loss scale imbalance affects optimizing language model quantization

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

Computation and LanguageArtificial Intelligence

Summary

Quantizing large language models (LLMs) helps make them run faster and use less memory, but it can be tricky to keep their accuracy. The authors found that the usual way to measure error during quantization, using mean squared error (MSE), causes uneven optimization across different model parts because it creates very different error levels at each stage. They discovered that using root mean squared error (RMSE) instead balances out these differences, leading to better and smoother optimization. This finding can improve how LLM quantization methods are designed and applied.

What this means in practice

  • For machine learning engineers: Improve the quantization of large language models by replacing mean squared error with root mean squared error to achieve more consistent optimization across model parts.
  • For ai hardware engineers: Design quantization workflows for transformer-based models that maintain model accuracy by balancing optimization strength through loss scaling techniques.

Authors

Chao Li, Shigeng Wang, Anbang Yao

Abstract

Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit quantized at each stage. State-of-the-art PTQ methods are predominantly learning-based, optimizing auxiliary quantization parameters (e.g., scaling factors, rotation matrices, clipping thresholds, and adapters) via gradient descent to minimize a reconstruction loss. A common practice is to use mean squared error (MSE) as the reconstruction loss function, yet its induced optimization behavior remains largely unexplored. In this work, we take a holistic view of sequential quantization and systematically investigate how optimization evolves from the first quantization stage to the last, aiming for a deep understanding of optimization in learning-based PTQ schemes. Through extensive empirical studies spanning representative learning-based PTQ methods, LLM families, model scales, architectures, quantization settings and various tasks, we consistently uncover Optimization Imbalance: reconstruction loss magnitudes vary dramatically across stages, accompanied by highly uneven gradient magnitudes and parameter updates under MSE. We term the cross-stage range of loss magnitudes the reconstruction loss scale, and reveal that MSE translates the unexpectedly large reconstruction loss scale into highly uneven gradient magnitudes, which in turn lead to uneven optimization strength across quantization stages. This finding suggests a general principle for improving learning-based PTQ: optimization strength across stages should be decoupled from the reconstruction loss scale. Theoretically, we show that root mean squared error (RMSE) variants defined at the sample, channel, token, and element levels naturally realize this principle through implicit gradient normalization, outperforming MSE significantly as a drop-in replacement.