Speculative decoding training reduces rounds in large language models
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Machine LearningArtificial IntelligenceComputation and Language
Summary
Making large language models faster can be done by using a smaller model to guess parts of the output quickly, and then checking those guesses with the big model. The authors studied how to train the smaller model better by thinking about the whole guessing and checking process, not just parts of it. They created a new way to measure how well the small model helps reduce the total checking done by the big model. Their method helps train these smaller models to do better at speeding up the bigger models in tasks like math, coding, and chatting.
What this means in practice
- •For ai inference engineers: Improve runtime efficiency of large language model inference by training draft models that minimize verification steps in batch decoding.
- •For automated code generation teams: Enhance the speed and reliability of code generation tools by applying improved training for draft models predicting multiple tokens at once.
Authors
Yunxiao Zhao, Changxiao Cai
Abstract
Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.