Adaptive memory management speeds up long context AI decoding
Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy
Artificial IntelligenceComputation and LanguageDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Long-context AI models struggle to handle all their memory efficiently, leading to slower performance. The authors created Adaptive DHiSparse, a system that adjusts which memory parts to keep readily accessible and which to fetch later, improving speed when working with very long text. This system also supports a guessing approach called speculative decoding that some AI models use to be faster. Tests show Adaptive DHiSparse can produce more output tokens per second and handle more simultaneous tasks compared to earlier methods.
What this means in practice
- •For ai platform engineers: Improve throughput and concurrency for serving long-context language models by adaptively managing memory hierarchies and supporting speculative decoding.
- •For cloud service operators: Increase efficiency of GPU utilization in large-scale AI inference by dynamically adjusting memory residency and reducing host fetches under variable load.
Authors
Naji Alazhar, Wassel Alazhar
Abstract
Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45$\times$ HiSparse and 1.27$\times$ non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4$\times$ the concurrency and 2.3 to 3.7$\times$ the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85$\times$ the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.