Agentic language model efficiency improved by action-guided memory management
ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
Operating SystemsArtificial Intelligence
Summary
Large language models (LLMs) that act step-by-step need to remember a lot during their thinking process, which uses a lot of computer memory and slows things down. The authors created ActKV, a method that smartly saves memory by keeping the most important parts related to future actions and throwing away less important details. It also changes how much memory it uses based on how confident the model is in choosing actions. This approach keeps nearly full accuracy but uses much less memory and runs faster.
What this means in practice
- •For ai system engineers: Build more efficient LLM agents that use significantly less memory while maintaining action reliability over long tasks.
- •For cloud service operators: Increase throughput for hosted LLM inference by adopting memory compression techniques tailored for action-driven workloads.
Authors
Zihan Wang, Cheng Tang, Lei Gong, Chao Wang, Wenqi Lou, Teng Wang, Xuehai Zhou
Abstract
Agentic LLM inference accumulates long KV caches across iterative observation-reasoning-action loops, imposing substantial memory overhead and limiting serving throughput. Existing compression methods emphasize overall output quality, overlooking the asymmetric importance of actions in driving task progress. Our key idea is to establish a compression criterion that values KV entries by their contribution to action generation and prioritizes action quality. However, iterative execution, dynamic memory demands, and scattered action-critical entries pose challenges to eviction policies, budget allocation, and paged memory integration. To this end, we propose ActKV, the first KV cache compression framework tailored for agentic LLM inference. (i) Action-oriented KV cache eviction exploits stable action access patterns to retain entries critical to future actions, supporting reliable task progress under compression. (ii) Confidence-driven adaptive budget allocation uses LLM's intrinsic confidence to adapt the budget to evolving action-critical memory demands. (iii) Page-aware compression management standardizes compression into three primitives with customized kernels, realizing practical throughput gains. On long-trace tasks, ActKV retains an average of 98.53% of FullKV's accuracy with only 25.98% of its peak KV cache memory. It also achieves 3.97 times and 3.58 times FullKV's token and task throughput, delivering state-of-the-art performance.