Lachesis improves cache memory lifetime for AI agent servers
Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash
Hardware ArchitectureDistributed, Parallel, and Cluster Computing
Summary
Serving AI language agents uses a lot of memory to store ongoing conversation context, which creates challenges for memory technology. The paper introduces Lachesis, a system that stores short-lived data in faster, smaller memory and long-lived data in larger, slower but durable memory. This approach extends the life of the storage hardware, especially high-bandwidth flash memory, by reducing wear from writes. The authors show through experiments that Lachesis can significantly prolong storage lifespan even under heavy, continuous use.
What this means in practice
- •For ai infrastructure engineers: Extend the lifetime of flash-based key-value caches in large language model agent deployments by managing data placement based on lifetime.
- •For cloud service operators: Maintain service reliability and hardware durability for AI agent workloads running continuously at full capacity by using lifetime-aware cache placement.
Authors
Jaehoon Yang, Jeongmin Lee, Haneul Park, Seung Yul Lee, Nam Sung Kim, Jae W. Lee
Abstract
Large language model (LLM) serving is increasingly dominated by agentic workloads, in which agents and their sub-agents accumulate context as KV cache across many requests, consuming substantial memory. High-bandwidth flash (HBF) is a promising solution, providing an order of magnitude greater capacity at HBM-class read bandwidth, but its finite write endurance is the key limiting factor. Our key insight is that KV cache should be placed across HBM and HBF by its lifetime. Placing shorter-lived data in HBM lets HBM absorb more of an agent run's writes and sends less of them to HBF. As the lifetime of KV cache in agentic serving is dictated by the harness, the program that orchestrates the agents, we analyze its behavior and identify three axes along which lifetime diverges, temporal, structural, and inter-worker. Guided by these observations, we present Lachesis, a lifetime-aware KV cache placement layer between the agent harness and the serving engine. At write time, it places each segment in HBM or HBF according to its lifetime, and frees its blocks once the segment is no longer read. In trace-driven simulation, Lachesis extends HBF lifetime by 1.19-3.13x over HBM-first placement, reaching 3.3-12.2 device-years. Even under continuous 24x7 operation at the full load a tight SLO admits, HBF outlasts its five-year warranty on the multi-agent trace.