Locally isolated rotation boosts large language model efficiency and speed
LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models
Artificial Intelligence
Summary
Large language models are powerful but require a lot of computing resources. The authors propose a method called LIRQuant that makes these models use less memory and run faster without losing accuracy. It carefully adjusts only the parts of the model that are sensitive to errors, rather than treating all parts equally. This approach leads to faster setup times and lower memory use when preparing the models for deployment, and speeds up their inference as well.
What this means in practice
- •For machine learning engineers: Calibrate large language models faster and with less memory to deploy efficient low-bit versions in production.
- •For infrastructure teams: Accelerate model inference and reduce memory footprint for large language models in cloud or edge deployments.
Authors
Desheng Xu, Hanglin Li, Tianchen Song, Zhiyong Zhao, Kun Zhan
Abstract
Post-Training Quantization (PTQ) enables efficient deployment of Large Language Models (LLMs), yet activation outliers remain a key challenge for low-bit quantization. Existing global transformation methods overlook heterogeneous channel sensitivity to quantization noise, leading to suboptimal accuracy-efficiency trade-offs. To address this, we propose \textbf{Locally Isolated Rotation Quantization (LIRQuant)}, a sensitivity-aware framework that selectively applies transformations. LIRQuant identifies sensitive channels using a noise-weighted sensitivity metric, applying expressive learnable rotations only to these channels and lightweight fixed rotations to the remaining ones. This decoupled design reduces optimization complexity without sacrificing outlier smoothing capability. Extensive experiments across the LLaMA-2/3, Mistral, Qwen3, and Mixtral families demonstrate that LIRQuant achieves state-of-the-art accuracy under the challenging W4A4KV4 setting. Meanwhile, LIRQuant calibrates LLaMA-3-8B in only 25 minutes with 20.3 GB peak GPU memory, achieving the fastest calibration and the lowest memory footprint among recent learnable-rotation methods, and its fused inference kernels deliver up to $2.3\times$ prefill speedup and $3.5\times$ memory saving over FP16.