Personal memory errors in language agents arise before text generation
Stale, Misattributed, or Late: Where Personal Memory Fails Before Generation
Artificial Intelligence
Summary
Language agents that remember facts about people often make mistakes before they even start talking. This can happen if their memory is outdated, mixed up between different people, or missing info when needed. The authors studied these memory problems directly and found that the hardest part is correctly matching updates to the right slots in memory. They also showed that some errors remain even when trying to filter them out before the agent answers. Their work suggests it’s important to check memory quality before generating any text.
What this means in practice
- •For chatbot developers: Improve chatbot accuracy by evaluating and correcting memory errors before generating responses.
- •For digital assistant teams: Reduce mistaken references in digital assistants by refining memory slot assignment and update processes.
Authors
Haonan Deng, Park Sinchaisri
Abstract
Personal memory for language agents is usually judged by whether the final an- swer is correct. That score hides errors that arise before generation: the memory block may contain an obsolete value, a fact about the wrong person, or no use- ful fact before the serving deadline. We measure these failures directly. Using Personal Fact Memory (PFM) as a reference layer, we find that temporal validity is primarily a property of memory construction in our setting. On a controlled revision benchmark, serving only the active value of each correctly keyed slot eliminates observed stale exposure; without update resolution, 70.3% of prompts expose a superseded value. Once retrievers share the same active store and par- ticipant information, participant-aware BM25 is equivalent to the reference ranker within a prespecified 0.02 margin. The harder problem is assigning revisions to the right slot. Missed merges leave stale values active, whereas false merges silently remove current values; four LLM key assigners achieve higher key re- call than a rule extractor yet produce lower clean-retrieval rates, and open-domain merge recall on LongMemEval never exceeds 0.062. Misattribution survives va- lidity filtering: an entity posterior reduces same-name exposure on controlled data but cannot distinguish identically named speakers in LoCoMo. Two frozen lan- guage models reproduce prompt errors in generated text. Retrieval latency varies across rankers, but prompt prefill dominates turn-level latency on our hardware. These results argue for evaluating agent memory before generation, separating stored-state validity, identity resolution, abstention, and serving latency.