Speculative decoding speeds up certain mixture-of-experts models with deduplication

Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length

Machine LearningDistributed, Parallel, and Cluster ComputingPerformance

Summary

Mixing experts in AI models can slow down the process when they repeatedly access the same expert parts. The authors study a way to speed this up by checking multiple guesses at once and only reading each expert once. They find when this approach is faster depends on how fast data can be read and how many tokens are checked together. Their method predicts the best number of guesses to try that saves the most time under different setups and hardware.

What this means in practice

  • For ai platform engineers: Optimize token decoding speed in MoE models by choosing draft length and deduplication strategies based on hardware bandwidth insights.
  • For gpu kernel developers: Improve throughput of expert operations in MoE models by implementing deduplication to reduce redundant data reads per decoding step.

Authors

Yuzheng Lan

Abstract

Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share $\bar u(n)$ of the per-slot bytes is below its ratio $r(n)$ of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.