LongSpark speeds up language model decoding without extra context cost
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Machine LearningArtificial Intelligence
Summary
Generating text with language models can be slow because each new word depends on all the previous ones. The authors found that the part of the system guessing multiple words ahead (the drafter) doesn’t need to consider all earlier words, unlike the main model that produces the final text. They designed LongSpark, a new method that keeps the drafter’s work fixed and small, no matter how long the text is. This lets it generate long passages faster without losing accuracy.
What this means in practice
- •For ai service providers: Run large language model services more efficiently by reducing the drafter’s memory and compute needs on long texts, speeding up responses.
- •For natural language processing engineers: Integrate LongSpark’s fixed-cost drafter to improve the speed of text generation for applications requiring long contextual inputs without extra computational burden.
- •For voice assistant developers: Enhance real-time speech-to-text systems by accelerating language model predictions over extended conversations using LongSpark’s efficient speculative decoding.$Commercial implications: Enables faster and more responsive voice assistants by lowering latency during long utterance processing, making it a sellable feature.
Authors
Hao-Yuan He, Peng-Fei Liu, Si Shen, Ming Li
Abstract
Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.