Agent memory in LLMs can confuse current tasks and a fix reduces this risk
Understanding and Mitigating Inference-Time Overreliance Using Agentic Memory
Artificial Intelligence
Summary
Sometimes, large language models (LLMs) that remember past experiences misuse those memories when solving new problems. The authors found that this problem happens especially when only parts of old memories match the new task, causing confusion. They created a method called MEMTRIM to clean up memory usage by removing repeated or conflicting information without needing to retrain the model. This approach helps the models rely less on misleading memories while still benefiting from helpful past knowledge.
What this means in practice
- •For software engineers: Improve chatbot reliability by reducing confusion from conflicting past memory during question answering.
- •For ai system integrators: Enhance LLM agent systems that reuse memories by applying MEMTRIM to filter stored knowledge at read and write time.
Authors
Luoxi Tang, Yuqiao Meng, Nilesh Auradkar, Muchao Ye, Dazheng Zhang, Zhaohan Xi
Abstract
Agentic memory allows LLM agents to reuse past experience, yet retrieved memories can also distort inference even when they are benign, correctly stored, and appropriately retrieved. We study this failure mode, which we call memory over-reliance. Across benchmarks and memory architectures, we find that memory is useful when past experience transfers to the current task, but can become misleading when only part of the evidence transfers. Failures are strongest under partial query-memory overlap, a pattern further confirmed by controlled experiments thatvary the amount of overlapping evidence. Motivated by this finding, we propose MEMTRIM, a plug-and-play framework that indexes memory evidence at write time and controls its reuse at read time. MEMTRIM removes repeated or conflicting evidence while preserving useful memory-specific information, requires no retraining, and applies to both embedding-based and structured memory systems.Experiments show that MEMTRIM reduces memory overreliance while preserving the benefits of useful memory across models and memory settings.