SharpDraft speeds up long-context text generation with query scaling
SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling
Artificial Intelligence
Summary
Generating long pieces of text using AI can be slow because the computer has to consider lots of context. The authors found that when the AI looks at more context, it spreads out its attention too thin, making draft predictions less confident and slowing things down. They created SharpDraft, a method that adjusts the way the AI focuses on key parts of the context to keep predictions sharp, without extra training. This approach significantly speeds up text generation while using the same amount of memory.
What this means in practice
- •For ai model engineers: Speed up long-context language model inference by improving draft token verification without increasing memory usage.
- •For cloud service providers: Reduce computational costs of running large language models on extended text inputs by integrating SharpDraft’s query scaling technique.
Authors
Jeonghoon Park, Seongwoon Jo, Jongwon Lee, Taesik Gong
Abstract
Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentrated on the highest-scoring Keys can decrease. We introduce SharpDraft, a training-free method that counteracts this effect through cardinality-aware Query scaling, without the computational overhead of online adaptation. Under explicit assumptions, we derive an exact top-$k$ mass correction and deploy a closed-form fixed-slope approximation. Across AIME-26, GPQA-Diamond, and LongGenBench Diary, SharpDraft achieves $2.59$-$3.19\times$ geometric-mean end-to-end speedups over target-only autoregressive decoding when applied to DFlash, PARD, and EAGLE 3.1. With DFlash, it improves decoding speed and outperforms full-parameter and LoRA-based online adaptation in end-to-end speedup, while matching the unmodified drafter's reported peak allocated GPU memory.