Message-passing method improves language model quantization accuracy
MPQ: A Message-Passing View of Post-Training Quantization
Artificial IntelligenceMachine Learning
Summary
Computer programs called large language models need a lot of memory, so they are often compressed by turning their details into simpler numbers, which can lower memory needs. Existing compression methods make strict choices for parts of the model without considering nearby parts enough, which might miss better options. The authors introduce a new method called Message-Passing Quantization (MPQ) that keeps track of many possible choices together before picking the best one at the end. This approach reduces errors better than older methods in tests with models like Llama, Qwen, and Mistral.
What this means in practice
- •For machine learning engineers: Improve compression of large language models to reduce memory use while keeping accuracy during deployment.
- •For ai platform developers: Integrate message-passing quantization algorithms to offer smaller model sizes with better performance to customers.$Commercial implications: Enables selling more efficient AI services with compressed models that maintain quality, appealing to cloud providers and AI vendors.
Authors
Yanshuo Chen, Chenxi Liu, Reza Shirkavand, Heng Huang
Abstract
Post-training quantization (PTQ) is a key technique for efficiently deploying large language models, as it compresses weights into low-bit representations without retraining. Existing PTQ solvers, including GPTQ, LDLQ, and block coordinate descent, commit each weight block to a single codeword before its neighbors are considered, leaving near-optimal alternatives unexplored. In this work, we cast row-wise PTQ as posterior inference over discrete codebooks, in which existing solvers correspond to a zero-temperature, hard-decision limit retaining only the top codeword per block. We propose Message-Passing Quantization (MPQ), which preserves soft posterior information across blocks during inference and commits to hard codewords only at the output stage. MPQ first uses vector approximate message passing to compute approximate blockwise posteriors, then refines a hard-decision initialization by exact block coordinate descent. Across 2-bit lattice and 3- and 4-bit scalar quantization on dense language models from Llama, Qwen, and Mistral families, MPQ improves perplexity and zero-shot accuracy over strong hard-decision baselines, and composes favorably with preprocessing techniques such as activation-aware scaling and incoherence processing. Our implementation is available at https://github.com/poseidonchan/MPQ.