Large language model decoding sped up by smarter expert selection
BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding
Machine Learning
Summary
Large language models can be slow because they use many expert parts, and loading all these parts for multiple requests wastes time. The authors have created a way to predict which experts matter most for each batch of requests to avoid loading unnecessary experts. This method uses a simple predictor to estimate how much leaving out each expert would affect the outcome, improving speed and accuracy without retraining the model. Their approach works better than previous methods and shows big gains on several tested models.
What this means in practice
- •For large-scale ai service operators: Reduce latency and resource cost of serving large language models by efficiently selecting expert components during batch decoding.
- •For machine learning engineering teams: Improve acceleration and efficiency of MoE model inference without retraining by applying predicted removal cost to expert selection.
Authors
Ali Abbasi, Justin Shi, Soheil Kolouri
Abstract
Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferring model weights from accelerator high-bandwidth memory into on-chip SRAM. Mixture-of-experts (MoE) models reduce computation by activating only a small subset of experts per token, but this sparsity does not translate directly to batched decoding. Different requests select different experts; therefore, the combined active set across many concurrent requests can span a substantial fraction of the expert pool and require significantly more expert weights to be transferred. Most expert-reduction techniques make retention decisions independently for each token and therefore do not address this batch-level expansion. More recently, batch-aware methods have attempted to coordinate expert use across concurrent requests and reuse experts already fetched for the batch. Yet their selection criteria are based primarily on router rankings or expert statistics collected during calibration. Consequently, these criteria are not directly tied to the output error caused by dropping an expert, nor do they capture how an expert's contribution changes across tokens at inference time. We instead rank experts according to how much their removal would change the MoE-layer output. To apply this criterion during serving, we train a lightweight linear predictor during calibration that estimates the expert removal cost for each incoming token, and develop custom GPU kernels for cost prediction and expert selection. Across three MoE architectures, BASE improves the quality-efficiency tradeoff without retraining. On Qwen3-30B-A3B, it improves average accuracy by 29.5 points over the strongest baseline at comparable throughput under a tight expert budget. At a higher expert budget, it is 60% faster than dense inference while remaining within 0.4 accuracy points.