Llm agents retain key interactions to cut processing costs and boost success

FOCUS: Training-Free Decision-Preserving Context Compression for LLM Agents

Artificial IntelligenceComputation and Language

Summary

Long conversations with AI agents can get slow and less accurate because the AI has to remember too much. The authors show a way to pick only the important past interactions that really affect future decisions without any extra training. This method, called FOCUS, works during real use and can be added to any AI model without changing it. It reduces the amount of memory needed almost by half and makes the AI perform better on tasks.

What this means in practice

  • For llm developers: Deploy models that dynamically compress past interactions to lower inference costs without retraining or offline data.
  • For chatbot platform engineers: Improve chatbot responsiveness and task success by attaching causal compression layers to existing AI backends.

Authors

Shantanu Dixit, Anson Bastos, Xuchao Zhang, Chetan Bansal, Saravan Rajmohan

Abstract

LLM agents accumulate interaction histories that grow linearly with task length, causing quadratic inference cost scaling and performance degradation from attention dilution. Existing context-compression methods learn what to discard offline: by contrastively optimizing guidelines, distilling compressors, or training compression policies. This incurs a substantial cost. Further, the compression policy is learned a priori and is not dynamically conditioned on the evolving test-time trajectories. In this paper we ask a complementary question: Which past interactions causally shape the agent's future decisions? We recast context compression as a causal decision preservation problem over discrete interaction units and introduce FOCUS, a training-free context compression framework that operates entirely at test time. Our method requires no offline data collection or fine-tuning, and is architecture-agnostic, attaching to any closed-API frontier model as a modular compression layer. We evaluate FOCUS on diverse agentic benchmarks including API and tool-calling, QA, web domain and multi-turn dialogue. Our method establishes new state of the art performance, cutting peak context by up to 48% and dependency by 73% while improving task success by up to 8.9 percentage points over uncompressed execution.