Improving large language model efficiency with smarter quantization
G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
Computation and LanguageArtificial Intelligence
Summary
Large language models need a lot of computer memory and power to run, so reducing these needs without losing quality is important. The authors found that current methods to shrink these models by simplifying their numbers can be improved by combining two types of mathematical information and updating this info frequently during the process. They also created a way to keep the changes stable and prevent big errors. Their new method called G2PTQ helps keep the smaller model closer to the original's performance.
What this means in practice
- •For machine learning engineers: Reduce memory and computation in deploying large language models with improved quantization accuracy using G2PTQ.
- •For cloud service operators: Use G2PTQ to optimize large language model hosting by lowering resource consumption without degrading model quality.
Authors
Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai, Yanqi Hao, Feiyu Wang, Weidong Zhong, Zhuang Wang, Tong Yang, Xiangsheng Zhou
Abstract
Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G$^2$PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G$^2$PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that G$^2$PTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines. Code is available at: https://github.com/G2PTQ/G2PTQ.