Cache pressure can cause long-lasting failures in LLM agent serving
When Cache Pressure Becomes a Trap: Metastability in Agent Serving
Distributed, Parallel, and Cluster Computing
Summary
When programs that use large language models (LLMs) send ongoing conversations to the model, reusing earlier parts helps keep things efficient. The authors show that a sudden increase in activity can mess up this reuse and cause a kind of trapped bad state where the system stays slow even after the surge ends. This happens because sessions take longer, increasing cache misses and delays that feed back on themselves. They build a tool called Basin to manage these effects, improving throughput during surges by carefully controlling session priority and cache use.
What this means in practice
- •For cloud infrastructure teams: Manage LLM serving systems to avoid long-lasting slowdowns after traffic surges by controlling cache use and session admission.
- •For online service reliability engineers: Improve recovery speed of AI-powered online services affected by sudden demand spikes using adaptive priority and cache management strategies.
Authors
Chengzhi Lu, Boxi Zhou, Yanying Lin, Dmitrii Ustiugov
Abstract
LLM agents repeatedly send a growing conversation to a model, making prefix reuse central to efficient serving. We show that a brief surge can disrupt this reuse and trigger metastable failure in prefill--decode (PD) disaggregated engines: service remains degraded after arrivals return to a load the engine previously sustained. On SGLang, a ten-minute surge reduced goodput by about 88% over the following hour. Removing the residual burst sessions restored service within minutes. The sustaining mechanism is population-cache feedback. Slower calls prolong sessions and enlarge their aggregate context, or working set; the resulting cache misses increase recomputation and further delay departures. A fluid model identifies the resource conditions for this feedback and explains how cache capacity, tool time and base load affect vulnerability. Across 36 simulated deployments, it flags all 17 cache-first cases. Controlled simulations hold the post-surge state fixed and show that changing subsequent arrivals alone reverses the recovery outcome. Once the cache collapses, low-reuse departure capacity sets the scale of the load reduction needed for recovery. Guided by these dynamics, we build Basin, which gives each session a fixed priority, reserves their predicted peak contexts against an adaptive admission budget, extends the cache with idle decode memory, and sheds excess demand. On real H200 1P3D deployments under surges, including long-context SWE-bench Pro sessions, an arrival-time prototype achieves 1.20--1.70x the goodput of Continuum and 2.35--3.59x that of HiCache alone. In simulation, the predictor-driven policy delivers 1.20--2.05x the strongest baseline's goodput across six surge scenarios.