Speculative decoding speeds up sparse MoE models more than predicted at low batches
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
Machine LearningArtificial Intelligence
Summary
Decoding is how computers generate text step-by-step, but it can be slow. Speculative decoding tries to speed this up by guessing multiple steps at once and checking them together. The authors found that for certain large AI models called sparse mixture-of-expert (MoE) models, this guessing-and-checking works even better than people thought, especially when dealing with small amounts of data at a time. They discovered this is because these fine-grained MoE models reuse parts of their computations, reducing the workload. The paper also introduces a new mathematical way to predict this improvement accurately.
What this means in practice
- •For machine learning engineers: Optimize deployment of sparse MoE models to achieve faster inference speeds at low batch sizes using improved speculative decoding techniques.
- •For high-performance computing teams: Design GPU kernels and memory usage strategies to maximize bandwidth and leverage expert routing overlap for efficient MoE verification.
Authors
Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli
Abstract
Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.