Large language models quantized with better accuracy using weight decomposition
Quantization of Large Models with Weight Outliers and Sparsity
Machine Learning
Summary
Storing big AI language models usually needs a lot of memory because their 'weights' are high precision numbers. The authors found a new way to shrink these weights into fewer bits while keeping the model's performance good. They break down the weights into simpler parts including a sparse part to handle unusual values, and use a clever search method to find the best breakdown. This method works well on several popular large language models and allows smaller memory use without hurting how well the AI talks and writes.
What this means in practice
- •For machine learning engineers: Deploy large language models using less memory by applying improved weight quantization with minimal performance loss.
- •For data center infrastructure teams: Reduce memory and storage requirements for hosting large language models, enabling more efficient use of hardware resources.
Authors
Leonidas Tsigkounakis, Abraham Jaeger Mountain, Yilang Zhang, Georgios B. Giannakis
Abstract
Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contribution decomposes the original trained weights into a quantized matrix, a low-rank component, and a sparse matrix that accounts for outlier rows in the weights. Due to analytical intractability and the discrete structure of the decomposition, a simulated annealing search is employed to explore candidate decompositions while avoiding poor local minima. Numerical tests demonstrate the effectiveness of this approach across a broad suite of large language models, including Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen2.5 7B, evaluated on natural language generation benchmarks using 4-bit NF4 and FP4 quantizers. The novel method can markedly outperform direct quantization while incurring minimal memory overhead.