Large language models quantized better through error decomposition and transformations
Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals
Machine Learning
Summary
Large language models can be made smaller and faster by reducing the precision of their memory and calculations, but this is tricky because some rare large activation values cause errors. The authors find a way to break down these errors into parts that can be fixed by changing the model’s weights and parts that need special transformations. They explain why certain random rotations, sign changes, and scaling methods work, and they offer practical rules for using these techniques. Their approach speeds up models like Llama and Mistral without training gradients and matches other advanced quantization methods.
What this means in practice
- •For machine learning engineers: Reduce memory and speed up inference of large language models by applying error-aware quantization methods without retraining weights.
- •For embedded systems developers: Deploy quantized large language models on low-resource devices using practical guidelines for compensating activation outliers and scaling.
Authors
Yamato Narita, Issei Sato
Abstract
Post-training weight-activation quantization reduces the memory and inference costs of large language models, but aggressive W4A4 quantization remains difficult because activation outliers degrade effective quantization resolution. Although weight optimization, channel-wise scaling, and orthogonal rotation mitigate this problem, the error components they address and their relationship remain unclear. Using an exact decomposition of local weight-activation quantization error into an activation-guided weight compensation term and an orthogonal residual, we bound the residual using persistent channel-wise outlier and regular activation quantities. This decomposition clarifies which error components can be addressed by weight compensation and which require transformation design. We then use the residual bounds to derive practical guidelines for applying randomized Hadamard rotation, sign selection, and channel scaling. In particular, the analysis explains how random signs suppress constructive interference among persistent outlier channels, how sampling multiple sign patterns can improve transformation selection, and how second-moment balancing leads to an $L_2$ scaling rule while a further relaxation recovers SmoothQuant-style $L_\infty$ scaling. We evaluate these guidelines through backpropagation-free configurations across eight Llama and Mistral models, obtaining performance competitive with gradient-trained SpinQuant.