ElasticKV reduces GPU memory stalls for faster language model responses
Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving
Machine Learning
Summary
Large language models often slow down because their memory gets overloaded with data needed to keep track of the conversation. The authors found that by using a clever way to store some of this data in a simpler form during high memory use, the models can keep running smoothly without waiting for all the detailed information. They built a system called ElasticKV that changes how much detail is kept based on available memory, making responses faster. Their tests show big speed improvements with no loss in the quality of generated text.
What this means in practice
- •For ai infrastructure engineers: Run large language models with less GPU memory stall, enabling faster response times under heavy workloads.
- •For cloud service providers: Offer improved LLM serving capacity by dynamically adjusting memory use to sustain high concurrency without request delays.
Authors
Jiantong Jiang, Yue Yang, Peiyu Yang, Feng Liu
Abstract
Large language model (LLM) serving is increasingly constrained by the GPU memory consumed by key-value (KV) caches. Existing compression, eviction, and offloading techniques alleviate this pressure, but serving runtimes typically treat only the configured target KV representation as execution-ready. Under memory pressure, this target-only contract can turn KV shortage into request stalls and preemptions. We present ElasticKV, a mixed-fidelity KV runtime built on the observation that target fidelity need not gate execution. ElasticKV introduces a compact intermediate KV state, making fidelity a runtime-managed execution property. To realize this state in a paged serving runtime, ElasticKV combines (i) a pair-structured layout that turns fidelity reduction into reusable GPU capacity, (ii) a dual-mode attention backend that directly consumes the compact state while preserving the native target-only path, and (iii) pressure-aware fidelity management that adapts KV fidelity to memory pressure. Our extensive evaluation across diverse workloads, model families and scales, and GPU platforms demonstrates the effectiveness and generality of ElasticKV. Under high concurrency, ElasticKV achieves 3.8-4.0$\times$ lower time-to-first-token (TTFT) and 9.1$\times$ lower P90 TTFT than vLLM while preserving generation quality.