SpecStream boosts long-context AI model responses with less GPU memory
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
Hardware Architecture
Summary
Large language models often need a lot of GPU memory to handle long conversations because they store extra information as they talk. The authors found that current ways to move this information between GPU and CPU are slow and waste computing power. They created SpecStream, which starts checking parts of the response before all the stored data is fully ready, and keeps some work on the GPU to avoid delays. This new system runs faster and supports more users at once without needing extra GPU memory.
What this means in practice
- •For ai infrastructure teams: Deploy efficient long-context language model serving that supports more users with limited GPU memory by using SpecStream’s parallel verification and selective KV offloading.
- •For cloud service providers: Increase AI model throughput per GPU to reduce hardware costs and improve response latency in hosted AI services handling large context inputs.$Commercial implications: Improves hardware utilization enabling cloud providers to offer faster and more cost-effective AI services with less GPU resource demand.
Authors
Fei Li, Song Liu, Shiqiang Nie, Jinyu Wang, Weiguo Wu
Abstract
Speculative decoding reduces sequential Target model calls by verifying multiple tokens from the Draft model in parallel. Yet KV Cache growth limits long-context serving under constrained GPU memory. Offloading KV to CPU memory relieves this pressure. However, existing offloading schemes restore the full KV history before attention and fail to fully exploit the benefits of KV sharing across queries within a verification round. Existing parallel speculative decoding methods also overlook idle GPU compute capacity while memory-bandwidth-bound Target verification waits for historical KV. We present SpecStream, a speculative decoding system that begins verification without waiting for the full KV history to be restored and exploits compute bubbles during KV transfers for concurrent drafting on the same GPUs. It offloads only Target-committed history, keeping candidate rollback local to the GPUs. Each streamed KV chunk serves all queries in the round, with online softmax preserving full attention. Target-priority scheduling controls Draft concurrent execution under resource limits to constrain interference with Target verification. Experiments show that SpecStream maintains task quality close to SGLang speculative decoding while supporting more concurrent requests under limited GPU memory. Across different datasets, it achieves average throughput speedups of 1.41$\times$ and 1.32$\times$ over the offloading baseline for Qwen3 and InternLM2.5, respectively. Compared with parallel speculative decoding on separate Target and Draft GPUs, SpecStream improves output throughput per GPU by an average of 55.4%.