Nucleus speculative decoding improves AI text generation speed and efficiency
Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
Computation and Language
Summary
Generating text with AI can be slow because the computer checks every new word carefully. The authors created a method called Nucleus Speculative Decoding that lets the AI accept words that look very likely without strict checking all the time. This speeds up text generation while keeping the output good. Their method balances speed and accuracy better by being more flexible about which words to accept.
What this means in practice
- •For language model engineers: Speed up text generation in AI models by accepting more plausible tokens during decoding to reduce computation time without hurting output quality.
- •For real-time chatbot developers: Improve responsiveness of conversational AI by generating responses faster using relaxed verification rules that still ensure plausible outputs.
Authors
Shuhao Li, Fanghua Ye, Wanyu Lin, Tianyu Yuan, Xiaoyu Shen
Abstract
Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distribution correction and rejects tokens that remain highly plausible under the target model when the draft model assigns excess probability. This conservative verification limits the number of draft tokens retained after each verification forward pass. We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding. NSD accepts a draft token if it satisfies the standard acceptance rule or belongs to the target model's nucleus. We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus. We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding. Experiments across multiple target models and proposal mechanisms demonstrate that NSD consistently improves speculative decoding efficiency while maintaining competitive task performance. Our method achieves throughput speedups of up to $5.16\times$ over autoregressive decoding and up to $3.15\times$ over standard speculative decoding. These improvements coincide with longer accepted lengths, allowing more output tokens to share the cost of each target verification pass. Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency. Our code is available at https://github.com/EIT-NLP/Nucleus-Speculative-Decoding.