MoE large language models keep quality with less storage using ProQuant
ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models
Artificial Intelligence
Summary
Large language models that use a mix of experts can reduce computing needs but still require a lot of storage. The authors find that current methods to shrink these models by lowering precision often lose quality because they miss certain errors and cause instability. They propose ProQuant, which smartly prunes parts of the model and adjusts precision to keep quality high and fix routing problems. Their experiments show ProQuant works better than older methods in producing good output from the models while saving space.
What this means in practice
- •For ml engineers: Compress MoE large language models with minimal quality loss to reduce storage and maintain performance in deployment.
- •For ai infrastructure teams: Optimize hardware resource usage by applying mixed-precision quantization strategies that keep routing decisions stable in MoE models.
Authors
Jiaqi Chen, Xianglong Yan, Chengzhu Bao, Xie Chen, Yulun Zhang
Abstract
Mixture-of-Experts (MoE) large language models reduce computation through sparse activation but still incur substantial storage costs. Mixed-precision post-training quantization (PTQ) reduces these costs by quantizing different model components at different bit widths, but existing methods often struggle to preserve generation quality under tight bit budgets. We identify two key challenges: (1) nonlinear quantization errors at extremely low precision are underestimated when cross-layer propagation is ignored; (2) direct router fine-tuning may cause routing drift, and calibration gains may not generalize to downstream tasks. To address these issues, we propose ProQuant, a weight-only PTQ framework for MoE models. Prune-or-Prioritize Mixed-Precision Allocation (PoP) jointly optimizes expert pruning and bit-width assignment. It allows higher-precision experts to substitute for severely degraded ones while improving bit-budget utilization. Routing-Consistent RMSNorm Finetune (RoCo) adapts expert input representations through mathematically equivalent transformations. It compensates for quantization errors to reduce routing drift. Extensive experiments demonstrate that ProQuant outperforms existing methods across diverse generative tasks. Code is available at https://github.com/ThrowBug/ProQuant.git.