PulseInfer speeds up large language model decoding with smarter memory use
PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
Machine Learning
Summary
Large language models (LLMs) use a memory system called KV cache during text generation, but storing all this data can slow down processing and waste GPU power. The authors found that moving most of this memory to the CPU and retrieving parts only when needed can cause slow data transfers and inefficiency. To fix this, they created PulseInfer, which schedules data transfers more cleverly, picks which data to move more efficiently, and combines small data pieces into bigger chunks. This approach makes the models decode text up to 4.7 times faster without losing accuracy.
What this means in practice
- •For machine learning engineers: Increase the speed of long-context language model decoding by managing memory transfers more efficiently between CPU and GPU.
- •For cloud infrastructure teams: Reduce GPU load and improve throughput when serving large language models by optimizing KV cache offloading and recall I/O.
Authors
Qiuyang Zhang, Kai Zhou, Kai Lu, Haocheng Lu, Jian Zhou, Yuanpeng Su, Kun Bao, Jiguang Wan, Fei Wu
Abstract
Long-context LLM serving is increasingly bottlenecked by decode, where large KV caches limit batch size and underutilize GPUs. Sparse KV cache offloading expands effective capacity by storing most historical KV blocks in CPU DRAM and recalling only selected blocks on demand. However, we find that existing offloading systems shift the bottleneck to CPU-GPU recall I/O: recall volume varies widely across layers, decode steps and requests, while headwise sparse selection fragments recalls into many small PCIe transfers. This paper presents PulseInfer, an I/O-centric sparse KV cache offloading system. PulseInfer hides variable recall latency with interruptible layer-wise scheduling, adapts offloading decisions with IO-Adaptive Offloading Admission, and coalesces fragmented transfers using SoloHead sparse selection and a gather-scatter I/O engine. Implemented on SGLang, PulseInfer improves decode throughput by up to 4.7x over SGLang and 2.6x over the best existing offloading baseline, while reducing TPOT by up to 76% and preserving near-lossless accuracy.