Parallel speculative decoding speeds up large language model output without extra memory
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Machine Learning
Summary
Large language models can generate text faster by guessing multiple upcoming words while double-checking those guesses with a more accurate model. The authors found that storing extra memory to help these guesses slows things down as many requests happen at once. They came up with a method that reuses memory cleverly without needing a separate cache, keeping quality high and speeding up output. Their method, called H-Spec, shows consistent improvements in how much text it can draft correctly and how fast it responds, especially under heavy usage.
What this means in practice
- •For cloud ai service operators: Improve throughput and reduce memory overhead when serving large language models concurrently with speculative decoding methods.
- •For natural language processing engineers: Integrate a hybrid memory reuse approach to speed up parallel token prediction without sacrificing draft quality in large model inference.
Authors
Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn, Reed Meyerson, Zhenting Qi, Tianyu Wu, Eldar Kurtic, Minlan Yu, Alexandre Marques
Abstract
Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification by the target model. Recent block diffusion drafters further reduce drafting latency by predicting multiple tokens in parallel. However, existing block drafters project target hidden states at every input position into a separate drafter-side KV cache, incurring per-request memory and KV-write overhead that grow with concurrency; directly reusing target KVs in place removes this cache but fails to sustain draft quality throughout the block. We propose a hybrid target-context injection method that complements direct target KV reuse with target hidden states only at the last input position, requiring no separate drafter-side KV cache. Building on this design, we propose H-Spec, a hybrid Mamba-attention parallel drafter that consumes the two target-context sources through complementary modules. Mamba modules are initialized with projected last-token target hidden states, while attention modules reuse target KVs in place. Despite its recurrent formulation, Mamba's parallel scan allows H-Spec to preserve block-parallel drafting. Across three target models and diverse tasks, H-Spec improves over the best baseline by 5.0--13.3% in mean accepted length and 5.3--12.6% in batch-size-1 inter-token latency speedup. Under concurrent serving, H-Spec consistently achieves higher throughput while maintaining lower KV cache utilization than baselines across evaluated concurrency levels.