Long-horizon agent reliability improves by pruning summary entries
Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents
Computation and Language
Summary
When AI agents work for a long time, they need to remember past interactions by summarizing them because their memory space is limited. The authors found that these summaries mix true facts with guesses from the summarizing step, causing many mistakes. They created a method to check each summary part to see if it actually comes from real events and remove parts that do not. This method is cheap, does not change the AI agent itself, and makes the agent more reliable without using more memory space.
What this means in practice
- •For ai system engineers: Improve AI agents that manage long conversations by reducing errors from inaccurate summary information without extra computational cost.
- •For enterprise software developers: Enhance office automation AI tools by increasing reliability when these tools summarize and act on long historical data interactions.
Authors
Guanghui Min, Chen Chen
Abstract
Long-horizon LLM agents rely on context compaction: when the interaction history outgrows the context window, a compressor replaces it with a natural-language summary. We show that these summaries contain more than a record of what happened. Compaction prompts ask for decisions and next steps, so each summary mixes facts established by the environment with reasoning written by the compressor, and the agent cannot tell the two apart afterwards. In an analysis of a widely used structured compressor, 83% of failed runs trace back to an action that carries out a specific summary entry, most of which no environment output supports. We propose Source-Judged Compaction (SJC), a training-free post-processing step that scores each summary entry for whether it has a source in the environment outputs or the previous summary, and removes the entries that do not. It changes neither the agent nor the compressor and adds one inexpensive judge request per compaction. The judge is Jev, a non-generative decision model that answers the source question with a calibrated probability instead of text. On AppWorld and OfficeBench with two agent models, SJC improves cross-run reliability over the underlying compressor by 2.1%-8.9% without increasing the agent's token usage.