TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization
2026-06-01 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors focus on making large language models use less memory and run faster by reducing the bit precision to 4 bits. Previous methods split model weights into two parts to handle precision loss, but they didn’t optimize how these parts behave after quantization. The authors propose TwinQuant, which learns better ways to split and adjust these two parts for improved accuracy and speed. Their method keeps the model's accuracy close to standard 16-bit precision and can run up to 1.8 times faster.
4-bit quantizationlarge language modelssingular value decompositionlow-rank approximationpost-quantization errorStiefel manifoldgeneral linear manifolddynamic rangeFP16 precisionend-to-end speedup
Authors
Haodong Wang, Junjie Liu, Zicong Hong, Qianli Liu, Jian Lin, Song Guo, Xu Chen
Abstract
4-bit quantization reduces the memory footprint and latency of large language model inference, but its aggressive precision reduction can severely degrade accuracy. Prior methods address this by decomposing each weight matrix into two components (e.g., via singular value decomposition) and quantizing them separately, assigning the bulk of values to a low-precision residual component while handling outliers with a high-precision low-rank component. However, such decompositions are designed to minimize the real-valued energy of the residual, rather than the post-quantization error of the residual and low-rank components. We propose TwinQuant, a 4-bit quantization framework that learns quantization-friendly decomposed subspaces and jointly reshapes both the low-rank and residual components. TwinQuant learns component-specific transformations via a joint optimization over the Stiefel and general linear manifolds, flattening their distributions and reducing dynamic-range imbalance. To enable efficient end-to-end execution, we further design a fused dual-component kernel that pipelines the two-stage low-rank computation on-chip and merges both components with a single epilogue, avoiding intermediate global-memory traffic. Across LLaMA3 and Qwen3 models, TwinQuant preserves near-FP16 accuracy and delivers up to $1.8\times$ end-to-end speedup over an FP16 baseline.