Carryover drafting boosts speed and token acceptance in large language models
Carryover Drafting: Recycling Rejected States for Speculative Decoding
Machine LearningArtificial Intelligence
Summary
Large language models can generate text faster by guessing multiple words at once and checking which guesses are good. Usually, when guesses are rejected, the work done to check them is thrown away. The authors found that keeping some information from these rejected guesses helps make better future guesses and speeds up the whole process. They created a new method called Carryover Drafting to reuse this information efficiently without slowing things down. Their experiments show it improves speed and quality of results in language tasks like translation.
What this means in practice
- •For ai model developers: Speed up large language model inference by reusing rejected token information to improve token proposal and acceptance rates.
- •For machine translation teams: Enhance translation system throughput by increasing acceptance length and reducing latency of decoding with Carryover Drafting.
Authors
Jahyun Koo, Sunghyeon Woo, Jaeeun Kil, Jeongtae Lee, Sungjae Lee, Kyomin Jung, Minsub Kim
Abstract
Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accepted and rejected tokens. Yet, conventional drafters retain only the representations of accepted tokens, leaving the substantial verifier computation spent on rejected tokens effectively wasted. We find that these discarded hidden states generated during target forward retain useful information about future tokens that can improve subsequent drafts. However, realizing this opportunity poses two distinct challenges. At inference, recycling overhead can increase drafting latency, diminishing the speedup gained from increased acceptance length. During training, standard parallel drafter training does not produce inference-aligned rejected states, while obtaining them through sequential rollouts would sacrifice parallelism across training positions. We introduce Carryover Drafting, which addresses both challenges. Carryover recycles rejected target hidden states as temporary KV context, allowing the drafter to selectively attend to them. It reuses the drafter's existing interface and adds only a single learned embedding to distinguish rejected states from committed context. The additional KV context is replaced each drafting round, keeping its length bounded by one proposal block. We introduce parallel draft--verify--draft training that exposes the drafter to inference-aligned rejected states while preserving parallelism across training positions. Experiments with DFlash and a DSpark-derived semi-autoregressive drafter across two target models show that this simple Carryover mechanism improves average acceptance length by 6.5--14.7% and end-to-end vLLM speedup by 7.9--14.4% over the corresponding baselines, with speedup gains reaching 28.8% on translation.