Long context language model decoding sped up with asynchronous self-speculation
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Machine LearningComputation and LanguageDistributed, Parallel, and Cluster Computing
Summary
Language models that read long texts are slowed down by repeatedly checking previous memory, which takes a lot of time. The authors introduce a method called ASPIRE that lets the model guess multiple words faster by updating guesses independently and checking them more flexibly. This approach mixes quick guesses with careful checks in the same step and refreshes memory during guessing to keep things accurate. Their tests show ASPIRE makes decoding 1.7 to 4.6 times faster than usual methods and improves speed compared to past guessing techniques.
What this means in practice
- •For ai infrastructure engineers: Increase throughput and efficiency of large language model services handling long input contexts by using asynchronous self-speculative decoding.
- •For cloud platform operators: Reduce computation costs and improve latency for hosting long context language models by integrating ASPIRE's mixed forward and scheduling approach.
Authors
Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao, Murali Annavaram, Salman Avestimehr
Abstract
Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviates this by drafting tokens with sparse attention and verifying them with full attention, but existing batched methods remain synchronized: all requests in a batch share a single draft-verify schedule, even though the optimal draft length varies widely across requests and changes dynamically within each request. We propose ASPIRE, a non-synchronized batched self-speculative decoding framework built on three components. First, a unified mixed forward allows drafting and verifying requests to coexist in the same batched forward pass, removing the need for global draft-verify phases. Second, a lightweight online speculation scheduler uses per-request acceptance-rate estimates and a batch-aware cost model to let each request independently choose when to verify. Third, an intra-draft refresh layer performs full attention at a single designated layer during drafting, updating the sparse context at every draft step to reduce staleness during drafting. Across three models and five reasoning and long-context benchmarks, ASPIRE achieves $1.70$-$4.58\times$ speedup in decoding throughput over autoregressive baselines and improves average speedup by approximately $27\%$ over the strongest prior self-speculative baselines.