Large language models quantized with better accuracy using weight decomposition

Quantization of Large Models with Weight Outliers and Sparsity

Machine Learning

Summary

Storing big AI language models usually needs a lot of memory because their 'weights' are high precision numbers. The authors found a new way to shrink these weights into fewer bits while keeping the model's performance good. They break down the weights into simpler parts including a sparse part to handle unusual values, and use a clever search method to find the best breakdown. This method works well on several popular large language models and allows smaller memory use without hurting how well the AI talks and writes.

What this means in practice

Authors

Leonidas Tsigkounakis, Abraham Jaeger Mountain, Yilang Zhang, Georgios B. Giannakis

Abstract

Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contribution decomposes the original trained weights into a quantized matrix, a low-rank component, and a sparse matrix that accounts for outlier rows in the weights. Due to analytical intractability and the discrete structure of the decomposition, a simulated annealing search is employed to explore candidate decompositions while avoiding poor local minima. Numerical tests demonstrate the effectiveness of this approach across a broad suite of large language models, including Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen2.5 7B, evaluated on natural language generation benchmarks using 4-bit NF4 and FP4 quantizers. The novel method can markedly outperform direct quantization while incurring minimal memory overhead.