Kvstream boosts large-scale llm cache retrieval with parallel storage
KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving
Distributed, Parallel, and Cluster Computing
Summary
When big AI models use long contexts, they reuse parts of previous data stored in a cache. But this cache gets too big for fast memory, so it has to rely on slower storage, causing delays. The authors designed KVStream, a system that smartly shares the work of fetching this cached data across different storage devices, speeding up access without needing complicated tuning. Tests showed KVStream can get data nearly as fast as theoretically possible, improving overall AI response speed, especially during heavy workloads.
What this means in practice
- •For llm infrastructure engineers: Improve large-scale LLM serving performance by reducing cache retrieval delays through adaptive parallel storage management using KVStream.
- •For cloud storage architects: Design multi-tier storage systems that balance workloads adaptively to optimize bandwidth usage for AI model cache access.
Authors
Avinash Maurya, Jaiaid Mobin, Jie Ye, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae
Abstract
Long-context LLM serving for agentic, coding, and document workloads reuses prefixes whose key-value (KV) cache outgrows GPU and host memory, putting storage I/O on the critical path of the inference step for retrieving prefixes. A single node-local NVMe absorbs all retrievals, while the parallel file system or cloud file store that nodes already mount carries almost no inference traffic. We cast prefix retrieval as aggregate-bandwidth scheduling across these tiers and present KVStream, a KV cache engine that batches a step's reads, defers writes behind the forward pass, replicates each chunk on every tier, and lets each tier adaptively claim the next unread chunks once it is free. Our performance model shows this routing attains the aggregate-bandwidth bound without prediction or tuning, whereas any static split misses it by a margin that grows as per-tier bandwidths drift or fluctuate. It also predicts that the time-to-first-token (TTFT) gain grows with load, as faster retrieval drains the queue, and disappears when host memory or arrival order keeps reuse in cache. Plugged into LMCache and evaluated on four models, six datasets, and up to 128 GPUs, KVStream retrieves within 5% of the bound, delivers up to 1.49$\times$ higher offline throughput than LMCache, and lowers mean TTFT by up to 6.1$\times$ under bursty arrivals.