Papers for

large language model engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Agentkv improves key storage for multitask language models

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Abstract: Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Mon 14 SeptMachine LearningComputation and Language
The gist
Language models that act with multiple steps, like thinking, acting, and using tools, need to remember many parts of their past interactions. The usual ways they decide what to keep or discard from memory assume future actions look like recent ones, but that doesn’t work well for these multi-phase tasks. The authors show that different task phases use different types of memory and offer a method, AGENTKV, that keeps track of these separately. This method improves task performance and speeds up the model’s output compared to previous approaches.
Open 2609.14872v1