Gpu cpu and ssd caching choices cut memory costs for ai sessions
Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions
Artificial Intelligence
Summary
Large AI models running chats and document answering keep track of past interactions using memory, but fast GPU memory is limited and expensive. The authors studied how best to store pieces of this memory across GPU, CPU, and SSD to support many sessions cheaply. They tested different ways to decide which memory parts stay where and when to move them using a detailed simulator. Their findings show smart memory tiering greatly increases session capacity and cuts cost, but complex placement policies and prefetching add little benefit. Simpler recency or reuse frequency rules save most data movement depending on the task.
What this means in practice
- •For ai infrastructure engineers: Optimize memory tiering strategies to increase concurrent AI sessions per GPU and reduce per-session hardware cost in multi-tier storage systems.
- •For cloud service operators: Design cost-efficient large language model hosting by prioritizing simple recency or reuse frequency policies over complex prefetching in multi-memory caches.
Authors
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
Abstract
GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD. The harder question is which blocks belong in each tier, when to move or evict them, and whether prefetching helps. We study these choices in a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. We compare recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads. Tiering supports 73.02 times more concurrent sessions per GPU and lowers cost per session by 62.04 times. These gains come from tier capacities of 1 plus 8 plus 64, not placement policy. Decode is compute bound at batch size one in our setup, so placement barely affects throughput. It mainly changes PCIe migration traffic and time to first token. Recency produces 2.30 times less migration traffic than reuse frequency for chat. Reuse frequency performs best for agents and document question answering. The existing predicted reuse policy is byte identical to recency, making its agent recommendation effectively recency. A genuine EWMA predictor changes behavior but still ranks behind reuse frequency on the workloads prediction was expected to help. Prefetching does not justify its bandwidth cost. Across the policy and cache size grid, even an oracle with knowledge of future requests never beats no prefetch on migration traffic. Workload specific placement can reduce data movement, but the predicted reuse and prefetch recommendations are not supported as implemented.