Dynamic expert skipping often underperforms simple static expert reduction
Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs
Machine LearningArtificial Intelligence
Summary
Many advanced AI models use groups of specialized parts called experts to understand information. Some new methods try to speed up these models by skipping experts dynamically for each piece of data, hoping to keep accuracy but use less computing power. The authors compared several of these complex skipping methods against a straightforward approach that reduces the number of experts used evenly for all data. They found the simple method often matched or beat the more complicated ones, suggesting simpler approaches may be better at saving computing resources without losing quality.
What this means in practice
- •For machine learning engineers: Optimize inference efficiency in large models by statically reducing active experts instead of complex dynamic skipping methods.
- •For ai system architects: Design multimodal AI systems with calibrated expert usage based on static expert count to ensure predictable compute costs during inference.
Authors
Jakob Steimle, Thomas Elsken, Daniel Ehrhardt, Lukas Rinder, Daniel Cremers, Xi Wang
Abstract
A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per token during inference, aiming to preserve model quality while further reducing the number of active parameters. These methods have added calibrated importance scores, inter-expert similarity metrics, or learned per-token allocators in order to identify experts to skip. However, different calibration regimes, different reductions in the effective number of experts, and evaluation suites make comparisons difficult. We classify four representative prior methods into a unified taxonomy and benchmark them against a natural, simple baseline: statically reducing the number of experts $k$ (i.e., same reduced $k$ across all layers and tokens). We provide a fair comparison of these methods at matched expert-compute budget, on two modern MoEs (Gemma4-26B-A4B and Qwen3-30B-A3B) in a text-only setting and a contemporary vision-language model (Qwen3-VL) in a multimodal setting. Surprisingly, we find that the simple baseline is hard to outperform consistently across models, modalities, benchmarks, and calibration settings. The methods we benchmark often underperform the baseline and achieved gains, when present, are minor. We further find that the effective $k$ achieved (or optimized for) in calibration can diverge from the effective $k$ in testing, resulting in over- or under-estimating the compute at inference time. Based on these findings, we recommend a calibration-free per-modality static $k$ reduction and validate this on a held-out model and modality. We further argue that a static control at matched measured compute is the baseline any dynamic skipping method must beat.