Speculative decoding improves throughput when acceptance gains outweigh costs
When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding
Computation and Language
Summary
Speculative decoding helps language models generate text faster by guessing parts of the output ahead of time. The authors studied when making these guesses more accurate actually leads to faster results overall. They found that improving guess accuracy can be cheap or expensive, and whether it's worth it depends on how uncertain the task is and the conditions of serving the model. Their work helps guide choosing and designing fast decoding methods based on specific needs and costs.
What this means in practice
- •For ml infrastructure engineers: Optimize language model serving by balancing acceptance improvements against execution costs to achieve better throughput.
- •For natural language generation developers: Select and tune speculative decoding strategies based on task uncertainty and serving environment to improve text generation speed.
Authors
Yongchao Long, Chunrui Zheng, Yuxuan Zhou, Guangkun Nie, Tian-Chi Ma, Chao Pang, Shuohuan Wang, Yuchen Ding, Xianbin Wen, Dianhai Yu, Junyuan Shang, Yuxi Zhou, Shenda Hong
Abstract
The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency. We ask when additional acceptance is worth the cost of obtaining it. We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six drafter recipes on shared target-generated data across ten benchmarks and two target models. Alignment changes the picture: gaps that appear large in published comparisons shrink substantially under shared training and inference conditions. This narrowing motivates two deeper questions: how is acceptance improved at different costs, and when do these gains improve throughput? We find that acceptance can be improved through nearly free objective and context changes or more expensive token-to-token conditioning. DCore illustrates the former route as a strong baseline with little additional measured drafting overhead. The payoff is conditional: task uncertainty and serving conditions affect acceptance returns, execution costs, and the preferred drafter and budget. These findings inform future drafter design and deployment choices.