SpectrumQuant improves low-bit model compression with adaptive rotations
SpectrumQuant: Spectrum Alignment for LLM Quantization
Machine Learning
Summary
Quantizing large language models helps run them faster and on smaller devices by reducing the bits used for numbers inside the model. The paper studies how to rotate data inside the model to make quantization more accurate without retraining. The authors introduce SpectrumQuant, a method that uses information from example data to choose these rotations efficiently and without heavy computation. Their approach balances using fixed rotations and fully learned ones, offering speed and memory advantages while keeping good performance.
What this means in practice
- •For machine learning engineers: Reduce memory and computation needs to deploy large language models on limited hardware using efficient post-training quantization.
- •For edge ai developers: Implement faster and less resource-intensive quantization methods for AI models running on edge devices without sacrificing accuracy.
Authors
Dwaipayan Haldar, Indranil Patra, Azhar Yousuf, Chandra Sekhar Seelamantula
Abstract
Orthogonal transforms have emerged as an effective tool for low-bit quantization of large language models (LLMs), as they reshape activation distributions without changing the underlying model function. Existing approaches typically rely either on fixed data-agnostic transforms, such as Hadamard rotations, or on calibration-time optimization of learned rotations. We study rotation selection from first principles and show that maximizing covariance leads to spectral alignment and provides a principled criterion for constructing quantization-friendly orthogonal transforms. Based on this insight, we propose SpectrumQuant, a closed-form rotation method that estimates activation covariance from calibration data and uses it to construct a covariance-aware orthogonal transform for post-training quantization. Our formulation characterizes quantization-friendly rotations as compositions of alignment and mixing components, while recovering Hadamard matrices as a special case. Interestingly, a class of butterfly matrices belongs to this category. In the realm of LLM quantization, SpectrumQuant bridges the gap between data-agnostic structured rotations and learned rotations, providing an effective and efficient data-aware alternative for post-training quantization. Experiments on Llama and Mistral models show that SpectrumQuant is competitive with state-of-the-art rotation strategies while requiring less than one-third of the GPU memory and significantly less time than learned rotations approaches like SpinQuant and DartQuant.