Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs
2026-08-31 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors developed Q-Strata, a new method to efficiently decide how many bits to use for each part of large Mixture-of-Experts (MoE) language models to keep performance high while keeping the model small. Unlike previous methods that handled bit allocation separately for each block or used simple proxies, Q-Strata considers the whole model at once and uses a clever two-step approach to find the best allocation. Their tests show Q-Strata performs better in reducing errors than current leading methods on various MoE models. This helps in making large language models smaller without losing too much quality.
Mixed-precision quantizationMixture-of-Experts (MoE)Large Language Models (LLMs)Bitwidth allocationPareto frontierQuantizationModel-level objectiveWikiText2 perplexityGPTQModel compression
Authors
Deokjae Lee, Sihun Chu, Hyun Oh Song
Abstract
Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-induced quality loss under a fixed budget, but Mixture-of-Experts (MoE) models contain these layers in every expert of every MoE block, so the allocation space grows far larger than in a dense model. Existing methods either allocate within each block under a uniform per-block budget, or allocate across blocks through an additive proxy, and neither directly optimizes a model-level objective over the choices that couple the blocks. We propose Q-Strata, a bi-level allocator that ranks within-block assignments with a cheap proxy and allocates across blocks with a model-level objective evaluated on the assembled quantized model. Its inner stage caches a Pareto frontier of candidates per block over finely spaced budgets, leaving the outer stage to set one budget per block instead of a bitwidth for every linear layer. With the search reduced to one budget per block, the outer stage optimizes this model-level objective directly, capturing the inter-block coupling that additive proxies miss. On Mixtral-8x7B-Instruct, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite, Q-Strata consistently achieves lower WikiText2 perplexity than uniform-bitwidth GPTQ and the state-of-the-art MoE MPQ methods MxMoE and GEMQ in the low-bit regime. The code is available at https://github.com/snu-mllab/Q-Strata/tree/main.