LoopQuant improves low-bit efficiency for looped transformer models
LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers
Machine LearningArtificial Intelligence
Summary
Looped Language Models reuse the same parts repeatedly to save memory, but this repetition makes it hard to shrink them into small, low-bit formats without losing accuracy. The authors present LoopQuant, a new method that carefully adjusts how these models are compressed by understanding their looping nature. This approach keeps the models accurate, reduces memory use significantly, and speeds up their operation compared to standard methods. Their tests show it works well across different looped models, making it easier to run these advanced models on devices with limited resources.
What this means in practice
- •For machine learning engineers: Deploy looped transformer language models on edge devices with limited memory by using efficient low-bit quantization without large accuracy loss.
- •For cloud ai service teams: Speed up inference and reduce memory costs for looped transformer-based NLP services by applying loop-aware quantization techniques.
Authors
Zhixiong Zhao, Zukang Xu, Xing Hu, Lifeng Liu, Guangyu Sun, Dawei Yang
Abstract
Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over evolving recurrent states poses new challenges for low-bit deployment. Existing post-training quantization (PTQ) methods, largely designed for feed-forward Transformers, fail to account for cross-loop activation dynamics, heterogeneous loop sensitivity, and recursive error propagation, leading to substantial performance degradation at low precision. To address these challenges, we propose LoopQuant, a loop-aware PTQ framework for accurate and efficient low-bit quantization of LoopLMs. LoopQuant is built on three core components: 1) Cross-Loop Distribution Canonicalization (CLDC) learns a shared quantization-friendly representation across recurrent states while preserving parameter sharing; 2) Interaction-Aware Loop-wise Precision Allocation (ILPA) allocates activation precision under a fixed bit budget by modeling task-level loop sensitivity and cross-loop interactions; and 3) Loop-Conditioned Task Calibration (LCTC) further aligns quantized inference with the low-bit recursive trajectory through lightweight task-space calibration. Extensive experiments across LoopLM architectures demonstrate that LoopQuant consistently outperforms existing low-bit PTQ methods. On Ouro-2.6B, LoopQuant under W4A4 surpasses all W4A8 baselines in performance, while reducing model memory by 65.3% with only 3.4% additional storage overhead over naive INT4 and achieving a 1.7 times decoding speedup over BF16. These results demonstrate a strong balance among accuracy, compression, and inference efficiency, providing a practical and deployment-friendly solution for low-bit LoopLM inference. The code is available at [https://github.com/Kishon-zzx/LoopQuant].