Eviction causes lasting memory errors in AI agents under token limits
What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory
Computation and LanguageArtificial IntelligenceDatabases
Summary
AI agents that remember past information have to forget some when their memory gets full. The authors studied which mistakes happen because the agent truly lost information, versus those that can be fixed by finding the right facts again. They tested several methods for deciding what to forget and found many errors are irreversible once memory is overwritten, especially with smaller memory sizes. This helps understand why AI agents mess up when forced to forget old info and how different forgetting methods compare.
What this means in practice
- •For ai system developers: Select eviction policies that reduce irreversible knowledge loss in AI agents handling long contexts.
- •For chatbot platform engineers: Diagnose and improve memory management to maintain accuracy when chatbots forget past conversation details.
Authors
Chen Shen
Abstract
Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the resulting loss in accuracy, but do not distinguish irreversible losses caused by eviction from recoverable retrieval failures. We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context and reruns the same reader. Combining the change in correctness with whether the evidence was retained after eviction classifies each oracle-answerable error as recoverable, irreversible, or residual; in the residual case, the answer remains incorrect after restoration. We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes, using GPT-4o-mini as the primary reader and judge and GPT-5.4-mini as a robustness reader. Under top-k retrieval at an 80k-token budget, the irreversible share among errors corrected by restoration is 0.67-0.73 for FIFO, random, and redundancy-aware eviction, compared with 0.60 for LLM-importance. At 8k tokens, it reaches 1.00 for all four policies. Recoverable errors occur under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not directly comparable unless the retrieval regime is reported. An exploratory matched-accuracy analysis detects no difference in irreversible rate among accuracy-matched policy pairs at a resolution of 1.2-6 percentage points. The same analysis detects the deliberately destructive control. To our knowledge, this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores on a standard conversational benchmark.