Speculative decoding energy savings vary with batch size and token acceptance
Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine LearningPerformance
Summary
Running large language models costs energy, especially when the model reads its own weights multiple times during word prediction. Speculative decoding tries to save energy by predicting several words at once and checking them together, but this saving depends on how many predictions get accepted. The authors find that when many tokens are accepted, energy savings happen, but at large batch sizes or with low acceptance rates, speculation can actually waste energy. They also show that these results change with the type of language and model used.
What this means in practice
- •For datacenter operators: Optimize server configurations by balancing batch size and speculative decoding to reduce energy per token under strict latency constraints.
- •For cloud service engineers: Design language model serving pipelines that maximize energy savings by tuning drafting acceptance rates and batch sizes for different languages and model types.
Authors
Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein
Abstract
Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.