Kv-cache offloading in llm serving trades predictability for latency

Characterizing Predictability-Latency Trade-offs of KV-Cache SSD Offloading in LMCache for LLM Serving Systems

Operating Systems

Summary

LLM serving systems often save GPU memory by storing large key-value cache data on SSDs, but the way these SSDs behave has not been well understood. The authors study two versions of LMCache software with different storage techniques to see how they influence read/write behavior and speed. They find that newer techniques trade stable and predictable response times for sometimes slower average performance. This helps system builders choose between consistent speeds or faster responses depending on their needs.

What this means in practice

  • For llm service operators: Design storage strategies balancing predictable response times and average latency for efficient large language model deployment.
  • For storage system engineers: Optimize SSD caching layers for workload patterns involving repeated or unique context prompts in LLM caching systems.

Authors

Ying He, Dingsen Shi, Yanbo Dai, Donger Chen, Harry Fu, Chris Tsu, Wang Feng, Song Fu

Abstract

KV-cache offload is widely used to stretch GPU memory for LLM serving, but its storage behavior has not been characterized at the block-device level. In this paper, we study LMCache through realworld multi-session workloads that span same/different context x same/different prompt, using over 100 stateless requests per workload. Our comparison is observational rather than factorial: it contrasts two realizable deployment snapshots -- LMCache 0.3.0 with buffered I/O and the kernel page cache, and LMCache 0.3.16 with prefix-aware deduplication and O_DIRECT. Key findings define the paper. First, the legacy stack appears almost read-free at the SSD layer in all four workloads, but a flush_ram experiment shows that this is conditional on page-cache warmth: once the cache is evicted, the same path issues about 1 GB of sequential reads in 683 ms. Second, deduplication delivers a dramatic write reduction only for the exact-repeat workload: one set collapses from 1.62 GB to 11.75 MB of writes, while the other three sets remain in the 2-5 GB range. Third, O_DIRECT converts a hidden and bimodal read cost into an explicit and stable one: on one set, warm TTFT rises from 27 ms to 93 ms because each of the 99 warm requests re-reads the same ~12 MB partial chunk from SSD. The main conclusion is therefore not "newer is better," but a system trade-off: predictability versus average-case latency.