Agentic language models use much GPU memory for context caching
Understanding KV Residency in Production-Scale Agentic LLM Workloads
Distributed, Parallel, and Cluster Computing
Summary
Large language models that act like assistants keep track of a growing conversation to answer users better. The authors studied how much memory these models need to store conversation context on GPUs during real work tasks. They found that keeping all context cached uses a huge amount of memory, far beyond what current GPUs hold. They tested techniques that reduce memory use by changing how context is stored or limiting how long it is kept, showing trade-offs between memory and speed. Their work helps design better systems for running these language models efficiently at large scale.
What this means in practice
- •For cloud infrastructure engineers: Optimize GPU memory use by implementing multi-resolution retention to reduce recomputation in large-scale language model services with long context workloads.
- •For ai system architects: Design deployment-aware routing or context migration to improve cache reuse and reduce recomputation across distributed language model deployments.
Authors
Mayank Arya, Agrim Bari, Anjaly Parayil, Yogesh Simmhan, Renee St. Amant, Victor Rühle, Saravan Rajmohan
Abstract
Agentic large language model (LLM) applications turn one user request into a sequence of model and tool calls over a growing context. Retaining this context's key-value (KV) state in GPU memory lets later calls reuse it instead of recomputing it, but at fleet scale this needs substantial memory. We characterize four production Microsoft M365 workload traces (Deep Research, Enterprise Assistant, Coding Agent and Enterprise Chat) spanning more than 121 million model calls. In Deep Research, the application issues 89% of model calls and the median context grows from about 10K to 68K tokens by the 30th call. About 80% of context tokens are served from the prefix cache. With the public Llama-2-70B KV layout standing in for confidential models, one 99th-percentile Deep Research session exceeds the 80 GB HBM of an NVIDIA H100, and holding all live sessions at their peak needs an estimated 501.8 TB of HBM, although inference occupies only 5.6% of that retention time. Trace-driven replay shows that a one-hour idle cap cuts this ceiling by 3.6x while raising the share of calls that recompute context only from 4.07% to 4.53%. Under tighter budgets, eviction alone is insufficient: at one tenth of the busiest hour's demand, LRU eviction makes 63% of Deep Research calls recompute and clairvoyant furthest-next-use 31%, whereas multi-resolution retention (MRR), which demotes the LRU session's KV state to int4, lowers this to 13.6%. Placement imposes a separate limit: when the next call reaches a different deployment, cached-context reuse drops from 89% to 29%, and treating KV state as deployment-local raises the recompute floor under unlimited memory from 5.4% to 20.2%. These findings motivate retention that changes the representation of KV state, together with deployment-aware routing or migration. We plan to release AgenticTraceSim, a parametric trace generator, and a residency replay framework.