Quantization moves perform differently depending on model state and order
Contextual Utility of Quantization Moves in Extreme Low-Bit LLMs
Databases
Summary
When compressing huge language models to use fewer bits, the impact of each small change depends on the current state of the model and the order in which changes happen. The authors show that measuring the effect of a change at its midpoint better predicts how it will affect model quality than just looking at the starting point. They also found that moves interact in complex ways, meaning that checking changes in isolation can be misleading. Their methods improve the accuracy of tiny-bit models by carefully evaluating these changes along their actual paths.
What this means in practice
- •For machine learning engineers: Improve accuracy of ultra-low-bit language models by refining how quantization steps are evaluated during model compression.
- •For hardware accelerator designers: Design quantization-aware optimizers for future hardware that better handle cumulative and interacting code changes in neural network weights.
Authors
Wenxuan Xiao, Xu Cao
Abstract
Post-training quantizers select finite code changes using reconstruction proxies or local loss approximations, but the utility of a quantization move depends on the state through which it is executed. We identify two sources of this contextual dependence. First, the displacement of the move matters: evaluating the gradient at the move midpoint captures curvature accumulated along the move that a current-state linearization omits. Across frozen two-bit moves from Llama-3.2 models, midpoint evaluation predicts the direction of exact endpoint loss changes substantially more accurately than current-state gradients. Second, moves interact: exhaustive lattices of legal quantized states are well approximated by quadratic pseudo-Boolean functions, yet their small pairwise components can determine Pareto fronts and cause different evaluation functionals to prefer opposite directions. These effects explain failures of reconstruction-optimal code re-selection and additive composition. Reading each move at its own midpoint repairs the local selection step and improves downstream accuracy and held-out perplexity, while larger supports require evaluating exact endpoints from the state actually reached. Exact-endpoint beam search finds sparse changes that dominate much larger one-shot updates, and repricing the same moves after intervening changes produces widespread sign reversals. These results show that quantization utility is contextual at the granularity of a few moves: reliable construction must evaluate finite changes along their own paths and compose them from the evolving quantized state.