Memory retrieval methods leave memory values uncertain in AI agents
Top-$k$ Retrieval Leaves Memory Value Undetermined
Artificial Intelligence
Summary
When AI systems decide which past memories to retrieve and use, they base this on logs of what was retrieved and whether the task succeeded. The authors found that common methods called top-k retrieval can’t actually tell which specific memories were valuable because the data doesn’t separate their effects. They created a new approach called LEDGER that clearly identifies only the memories whose value can be determined, improving how AI systems decide what to remember and use next.
What this means in practice
- •For machine learning engineers: Improve AI memory systems by using LEDGER to identify which memories truly impact performance and better guide retrieval strategies.
- •For natural language processing teams: Enhance question answering models by applying LEDGER to select and merge memories that add value, improving interpretability and accuracy trade-offs.
Authors
Arman Behnam
Abstract
A memory-augmented agent must decide which memories to keep, merge and retrieve, and its only evidence of their value is its retrieval log: the memories it retrieved in each episode and whether it succeeded. Existing methods assign every memory a value from this log, even when the log cannot separate one memory's contribution from another's. We treat the log as an experiment that nobody designed and ask which memory values it determines. Two counts answer this before any outcome is observed: how often each memory was retrieved, and how often each pair was retrieved together. From these counts, we prove that top-$k$ retrieval leaves every memory value undetermined, however many episodes are logged, and that a question with its own memories, asked once, determines at most one of their values. On fourteen standard memory benchmarks, no memory value is determined in any of 55 configurations. Both causes are choices about retrieval. We introduce LEDGER, which estimates each determined Shapley value in closed form with a confidence interval and reports every other value as undetermined. It merges memories with identical histories without losing a determined value, and it retrieves so that the next log determines more. On pooled HotpotQA, LEDGER raises the fraction of memories determined from 0.001 to 0.797, at a cost of 2.6 accuracy points on HotpotQA and with a gain of 2.7 on LoCoMo.