Activation denoising improves large language model quantization speed and accuracy

Activation Denoising: A Robustness View on Parallel vs Sequential LLM Quantization

Machine Learning

Summary

Large language models need to be compressed to run efficiently, but this compression can introduce errors that add up as data moves through the model. The usual way to fix this is to adjust each layer one by one, which is slow. The authors propose a new method that treats these errors as noise and trains the model to be robust against them all at once. This method achieves much of the quality of the slower approach while running much faster, helping make large models easier to use at scale.

What this means in practice

  • For machine learning engineers: Speed up quantization of large language models without losing much accuracy, enabling faster deployment on resource-limited devices.
  • For cloud ai infrastructure teams: Reduce the operational bottleneck in compressing large models by using a parallel-friendly quantization method that maintains model quality.

Authors

Yan Scholten, Rachel Lawrence, James Hensman, Stephan Günnemann, Alicia Curth, Riccardo Grazzi

Abstract

Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but quantization errors then compound through the residual stream, as no layer corrects for the errors of the layers before it. Sequential quantization accounts for this error compounding by re-calibrating each layer on the already-quantized outputs of its predecessors, yielding stronger results but at the cost of a serial schedule that becomes a bottleneck at scale. As a solution, we propose parallel quantization with activation denoising, which recovers much of the sequential benefit while keeping quantization fully parallel. Rather than re-calibrating layer-by-layer, we take a robustness perspective and model the upstream error as noise, regularizing to be robust to it through a preprocessing step followed by metric-weighted rounding. Applied at every layer, this regularization forms a depth-compounding smoothness penalty that dampens how strongly quantization errors amplify through the model. Unlike orthogonal rotations commonly used in quantization, which must preserve the model's function, we multiply the weights by a more general linear transformation. We find that the two are complementary and their effects compound. Empirically, our robustness regularization recovers a significant part of sequential quantization's benefit in a single parallel pass, at a fraction of its time. Overall, by treating compounding quantization errors as a robustness problem, we offer a principled foundation for more efficient and accurate LLM quantization at scale.