SAGE improves 2-bit quantization for faster large language models
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
Artificial Intelligence
Summary
Large language models are very powerful but need a lot of computer power to run. One way to reduce this cost is by squeezing their weight data into fewer bits, but using very few bits often leads to big errors. The authors propose SAGE, a new way to carefully group and encode the model’s data so that it keeps most of its accuracy even when squeezed to just 2 bits. This method also speeds up the model when running on GPUs, making advanced AI tools more efficient and practical.
What this means in practice
- •For machine learning engineers: Deploy large language models with reduced memory and faster inference using 2-bit quantization without large accuracy drops.
- •For ai platform operators: Run large language models more cost-effectively in cloud environments by leveraging SAGE’s efficient compression and custom GPU kernel.
Authors
Xianglong Yan, Zenan Wu, Kewei Zhang, Kaicheng Yang, Yulun Zhang
Abstract
Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better preserves accuracy through more expressive joint encoding. However, existing VQ methods still exhibit substantial degradation on real-world tasks, and two design challenges remain underexplored: (1) protecting sensitive weight channels within jointly encoded vectors; and (2) allocating a fixed bit budget among representation components, such as direction and radius, to suit different weight matrices. To address these challenges, we propose SAGE: Split Adaptation and Grouping by Energy for weight-only vector quantization. First, Energy-Stratified Grouping (ESG) couples channel grouping with activation-weighted fitting, grouping channels from different energy tiers within each subvector to prioritize sensitive channels. Second, Adaptive Bit Split (ABS) balances direction-codebook capacity and radius precision per matrix, selecting the split with the lowest activation-weighted reconstruction error under a fixed budget. Across benchmarks covering knowledge, mathematics, code generation, instruction following, and tool use, SAGE achieves state-of-the-art performance in the 2-bit quantization setting. On Qwen3-30B-A3B, it retains 95.3% of the BF16 model's average benchmark score. With our custom-designed CUDA kernel, SAGE delivers up to a 2.55-fold non-streaming end-to-end speedup over FP16. The code is available at https://github.com/Zincwolf/sage.