Context compression can reduce tokens but slow large language model agents
Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
Machine LearningArtificial Intelligence
Summary
Longer tasks for AI language agents need them to compress their past conversations and actions to fit into memory. The authors found that using fewer words doesn’t always make these agents faster or cheaper to run. In some cases, compressing the context heavily made the agent slower but still as successful. Their study shows that how, when, and how much you compress should be decided based on the specific task, AI model, and workload.
What this means in practice
- •For ai system engineers: Optimize memory and processing trade-offs in AI agents by tuning context compression strategies per task and model to balance speed and success.
- •For cloud service providers: Estimate costs and latency impacts of context compression in multi-model AI workloads to improve resource allocation and pricing.
Authors
Ritul Satish, Prasoon Sinha, Akiho Kawada, Neeraja J. Yadwadkar
Abstract
As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.