Low latency key value cache offloading boosts long context llm serving
FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
Artificial Intelligence
Summary
Large language models need to store lots of data during conversations, which can overload fast memory and slow down responses. The authors present FlashOffloading, a method that moves this data to slower but more plentiful memory without causing delays. They designed special caches and techniques to keep things running smoothly and faster. Their tests show it can speed up different parts of the language model’s work significantly while meeting strict timing goals.
What this means in practice
- •For llm serving platform engineers: Increase concurrency and throughput in long-context large language model serving by efficiently offloading key-value caches to host memory.
- •For high-performance computing teams: Optimize resource use and reduce latency in memory-intensive AI workloads through advanced cache offloading and parallel compute-load scheduling.
Authors
Xuan Wang, Zhuanhao Wu, Tingzhu Fang, Xushi Li, Sicheng Zhang, Zhenyu Yang, Feng Liu, Zhaohui Fan, Qiang Xie, Huayi Zou, Yuncong Liu, Zeyu Song, Ke Zhu, Yanjia Zeng, Jingchun Gao, Yan Shi, Kui Luo, Gezi Li, Liangjun Feng, Cheng Li
Abstract
In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidth memory (HBM) capacity wall and results in low concurrency. Offloading the KV cache to host dynamic random-access memory (DRAM) frees HBM space and raises concurrency, but the additional cache management and host-to-device (H2D) load overhead inflates inference latency, higher concurrency fails to translate into effective throughput. We present FlashOffloading, a host-DRAM-centric inference architecture that fully offloads the KV cache in both the prefill and decode phases. FlashOffloading builds rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index with zero extra miss-lookup overhead, and a compute--load overlapped kernel, optimizing KV offloading latency across the entire pipeline and delivering throughput gains under a time-per-output-token (TPOT) service-level objective (SLO). In experiments deploying GLM-5.2 with multi-token prediction (MTP3) speculative decoding on an Ascend910C prefill--decode (PD)-disaggregated platform, FlashOffloading improves prefill throughput by up to 1.38\(\times\) and decode throughput by up to 1.91\(\times\) under a 30 ms TPOT constraint. FlashOffloading is integrated into an open-source serving framework: https://github.com/vllm-project/vllm-ascend/tree/releases/v0.30.0.