Routing requests improves quality and efficiency of fixed quantized AI models
Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances
Machine Learning
Summary
AI models called Mixture-of-Experts can be stored in different versions using less memory, but this reduces their quality unevenly depending on the input. The authors study how to smartly send requests to these fixed stored versions to keep quality loss within a budget and make processing more efficient. They introduce a new way to predict the quality drop for each request and use a mathematical optimization to assign requests to model instances accordingly. Their approach improves throughput and reduces quality loss compared to simpler baseline methods.
What this means in practice
- •For ai service operators: Assign requests to pre-stored quantized AI model versions to maintain service quality while improving throughput under fixed memory constraints.
- •For cloud infrastructure teams: Optimize routing of varied workloads over fixed AI model replicas to reduce reconfiguration overhead and maintain expected quality levels.
Authors
Zhenghong Huang, Hongfan Wu, Jiheng Zhang
Abstract
Quantized Mixture-of-Experts (MoE) services can hold several pre-materialized instances of one base model, but quantization damage varies sharply across requests and bitwidths. Because instance materialization and replica counts consume memory and require slow reconfiguration, we treat them as upstream provisioning decisions and study routing within a fixed resident pool. Within this fixed-pool boundary, we route each request to maximize modeled throughput under a class-level expected quality-degradation budget and measured instance capacities. To predict this request-specific risk, we introduce FWP (Fragility-Weighted Perplexity), computed from prompt tokens on a reference-instance prefill and calibrated to candidate-instance degradation. Underlying FWP is an exact two-expert affinity--fragility decomposition and a conditional multi-layer top-$k$ expansion whose bias, interaction, route-change, separability, and higher-order terms remain explicit. Using these calibrated risks, a window-level linear program yields a signed reduced-reward score that is KKT-consistent with the LP optimum under optimal prices and primal-feasible tie allocation. On 88 extended Qwen prompts, complete W2, W3, and W4 instances quantizing all 6,144 expert blocks incur mean $Δ$NLL of $0.9437$, $0.1832$, and $0.0513$. Under the same population and $τ=0.1513$, FWP allocation reaches a $1.284\times$ offline model-based multiplier versus $1.253\times$ for request-agnostic mixing and $1.000\times$ for static W4, an incremental $2.5\%$ relative FWP gain.