Cache engineering improves large language model agent efficiency and speed
Cache Engineering for Agents
Computation and Language
Summary
Large language model (LLM) agents work on complex tasks by remembering long histories of their previous steps, but storing and using this memory can be slow and costly. The authors introduce "cache engineering," a way for these agents to better control what parts of their memory they focus on without retracing all previous steps or losing important information. They also created a system named AKV that manages this improved memory control behind the scenes, making the whole process faster and cheaper. Experiments showed that this approach both improves accuracy and reduces computation costs compared to older methods.
What this means in practice
- •For ai infrastructure teams: Reduce computation costs and improve response times of large language model agents by implementing advanced cache management policies.
- •For software engineers building conversational agents: Enhance long-term memory handling in conversational AI by using cache engineering to keep relevant history without costly recomputation.
Authors
Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, Yu Wang
Abstract
Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.