Memory store details determine if agents can judge data validity
Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It
Artificial Intelligence
Summary
Sometimes, a system with memory might accidentally use old or canceled plans when answering questions. This paper looks at what details a memory system has to provide so the agent knows not to trust certain records before it even looks at them. The researchers tested various language models to see how well they can understand this 'meta-memory' information based only on summaries of the data. They found that the quality of what the memory store writes down limits how well any model can make these judgments.
What this means in practice
- •For software engineers: Design memory backends to record critical metadata enabling agents to avoid outdated or invalid records before retrieval.
- •For machine learning developers: Improve language model readers by incorporating more detailed lifecycle state metadata from memory stores to boost judgment on data relevance.
Tested on simulated data.
Authors
Ansuman Mullick, Eray Tüzün
Abstract
An agent with long-term memory can answer from a record it should no longer use, such as a plan the user later cancelled. We ask what a memory store must expose for an agent to know this before retrieving anything, and we score that judgment on its own, as metamemory monitoring. Readers see only a value-free summary: record counts by lifecycle state and a list of attribute names. Each of 148 questions is asked against three versions of one store that differ in one attribute, so wording cannot give the answer away. What a model adds depends on the length of that list. On the short lists the benchmark's ground truth produces (2.6 names on average), none of five language models reliably beats a cosine-similarity lookup over the names, and the three stronger ones are equivalent to it within 0.05. Under a control that fixes counts and list length, none is reliably above it and none is shown equivalent. On lists longer than the stores we tested write, padded to 60 names, the lookup loses 0.18; GPT-5.6 Luna and Sol lose 0.06 to 0.07 and lead it by 0.13 to 0.14, while the other three fall with it. The lead holds for Sol against a sibling padded to the same length and counts. Reworded to share no word with the names, the lookup loses 0.02 and one stronger reader edges 0.04 ahead. On the short lists the three stronger readers lead only where the summary counts a state without naming it, and one count feature closes that lead within a question, though not when items are pooled. The backends we tested omit or misstate this information: under FR-Bank's own metadata, GPT-4.1 mini, Haiku 4.5 and the lookup fall from 0.73, 0.82 and 0.75 to 0.60, 0.71 and 0.59. In these tests, what the store wrote down limited every reader we gave it to. All stores are synthetic; control, whether a better judgment yields a better answer, is left to a later study.