FlatCake speeds up large language model memory with unified storage
FlatCake: A Flat KVCache Storage Architecture for LLM Serving
Distributed, Parallel, and Cluster ComputingOperating Systems
Summary
Large language models use a memory system called a key-value cache to run faster by remembering parts of previous work. Existing systems use two separate memory layers—fast DRAM and slower SSD—but the authors found this causes problems with new uses where memory needs are either very short or very large. The authors created FlatCake, which treats DRAM and SSD as equals in one memory space, putting each piece of data exactly where it belongs so it never needs moving. This design lets machines access memory faster and more efficiently, significantly improving speed and throughput in real large language model tasks.
What this means in practice
- •For large language model engineers: Serve large language models faster by using FlatCake's unified memory storage design to reduce latency and improve cache efficiency on real workloads.
- •For cloud infrastructure teams: Optimize server memory management for AI model deployments by adopting FlatCake’s flat KV cache architecture, improving SSD utilization and reducing read amplification.
Authors
Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, Youwei Zhuo
Abstract
KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-shot blocks with long reuse distances or by multi-gigabyte agentic sessions that exceed DRAM capacity, putting SSD on the critical read path. Second, DRAM staging and eviction-ordered layout bottleneck SSD restoration. Holding host staging buffers for in-flight prefixes causes contention under load that aborts prefetches, while incremental eviction scatters a session's blocks across the SSD and prevents large sequential reads. We present FlatCake, a flat KV storage architecture that exposes host DRAM and NVMe SSD as peer media in a unified 64-bit address space. Exploiting the fact that KV blocks are write-once, immutable, and reused at session scope, FlatCake places each block directly onto its target medium in session-contiguous, layer-major order at creation time and never migrates it. FlatCake streams coalesced SSD reads directly into a bounded GPU window via GPUDirect Storage and decouples DRAM and SSD request queues to prevent head-of-line blocking. On a production Claude Code trace with DeepSeek-V4-Flash, FlatCake sustains a 94% cache hit rate as load increases (versus 64% for Mooncake), eliminates Mooncake's $6.5\times$--$6.7\times$ SSD read amplification, and reduces peak-load mean TTFT by $1.6\times$ while improving total token throughput by $1.6\times$. Across 8K--128K contexts on Llama-3.1-70B, FlatCake further reduces mean TTFT by up to $2.4\times$ over Mooncake and $1.8\times$ over Mooncake-GDS.