Efficient agent memory use cuts waiting time for AI coding helpers
EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Sometimes language AI helpers redo work they've already done, which wastes time. The authors found that saving parts of the conversation in a special memory space on the computer helps, but only if this memory is big enough and managed smartly. They built EfficientAgent, which predicts how much memory is needed and decides what to save or skip. This approach reduces repeated work and speeds up responses, especially when the computer has less powerful GPUs. The results came from careful tests with AI coding agents.
What this means in practice
- •For ai platform engineers: Designing serving systems that manage memory efficiently to speed up multiple concurrent AI agents by smartly sizing and controlling cached context offloading.
- •For cloud gpu operators: Reducing response latency and GPU load during high concurrency workloads by adjusting cache offload strategies based on GPU compute and host bandwidth balance.
Authors
Kunming Shao, Jierun Chen, Jiangnan Yu, Xiao-Hui Li, Chaofan Tao, Yanli Wang, Huanxin Lin, Kwang-Ting Cheng, Chi Ying Tsui, Haoli Bai
Abstract
LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (KV) state and, when GPU memory runs out, by offloading that state to host memory. For agents, offloading gives inconsistent results: on the same coding-agent workload it speeds up one deployment, slows down another, and changes nothing on a third, even where loading a token back is several times cheaper than recomputing it. The reason is that cached state must survive until it is used again. While one agent waits for its tool, the server processes the contexts of all other agents, so an agent's prefix is reused only if the host tier holds the reusable context of the whole agent pool, which we call the reuse working set. A smaller tier keeps writing state that is evicted before anyone reads it. We present EfficientAgent, which sizes and manages the host tier by this working set. A stack-distance model estimates the working set from agent histories to size the host tier; its predictions, made before the experiments, located the capacity at which offloading starts to pay. When the tier is too small, a runtime policy stops writing large refills of evicted context and keeps extending prefixes that are still cached; when the tier is large enough, it writes everything. On SWE-bench Verified coding agents, a host tier sized to the estimated working set cuts recomputed prompt tokens by 93% and end-to-end time by 39%. With a small fixed tier, the policy cuts recomputation by 35%; with a large tier, it avoids the 4.3-fold increase caused by always filtering writes. Across three GPU types and two models, offloading pays off when the GPU has little compute per byte of host bandwidth and the host tier holds the working set. Code is available at https://github.com/KunmingSHAO/efficientagent_release.