Mixed precision approach improves storage efficiency in large models
Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
Machine Learning
Summary
Large AI language models called mixture-of-experts activate only a few parts at a time but must keep all parts stored, which takes a lot of space. Simply ranking parts to decide their storage precision doesn't fully capture how important their accuracy is. The authors present NatScale, a new method that adjusts how much storage precision each part gets based on a calibrated measure of its error impact, while respecting their importance order. Testing on several models shows this method better preserves model behavior with less storage than other ways of allocating precision.
What this means in practice
- •For machine learning engineers: Reduce storage requirements when deploying large MoE language models by allocating bit precision more effectively per expert module.
- •For data center infrastructure teams: Optimize memory usage in servers hosting MoE models to improve efficiency and lower hardware costs.
Authors
Renjie Xie, Juncheng Yang, Aoting Hu, Yan Liang, Mingxi Zhang, Zheheng Hong, Wei Xu, Yanzhi Chen
Abstract
Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.