Efficient rotation method speeds up large language model quantization
ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs
Machine LearningArtificial Intelligence
Summary
Quantizing large language models involves simplifying their internal numbers to make them run more efficiently, but this is usually slow and needs lots of data. The authors introduce ThinQuant, a method that picks fewer but smarter data points to learn better rotations for quantization. This approach drastically cuts down computation time without losing accuracy, allowing it to work on very large models much faster than before. ThinQuant also performs better than previous methods in tests measuring language understanding.
What this means in practice
- •For machine learning engineers: Optimize quantization calibration processes to reduce runtime for deploying large language models efficiently on hardware.
- •For cloud ai service providers: Accelerate model compression workflows to offer faster and cheaper hosted large language model inference services.$Commercial implications: Enables faster quantization calibration for large models, allowing service providers to reduce operational costs and improve deployment speed.
Authors
Mehdi Makni, Ryan Lucas, Rahul Mazumder
Abstract
Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (ii) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an $r$-dimensional subspace with $r<d$, making optimization over a $d\times d$ rotation equivalent to optimizing a $d\times r$ matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.