Memory shifts reveal detection and repair limits in ai assistants
MemShift: Separating Detection from Repair in LLM-Derived Personal Memory
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Keeping personal information correct in AI helpers involves noticing when something changes and fixing it. The authors show that traditional tests don’t separate these two tasks well. They created a new test called MemShift that checks if an AI notices changes and then repairs its stored info correctly. Their tests found that current AI models often struggle to fix changes even when they notice them. They also show that using programs over structured records improves accuracy more than relying on the AI alone.
What this means in practice
- •For software developers: Evaluate AI memory systems’ ability to detect and fix changed user data using MemShift’s detailed benchmark.
- •For customer support teams: Improve AI assistants by diagnosing failures in updating customer contact info through separate detection and repair evaluation.
Authors
Zichuan Xu, Haochen Zhang, Yuyang Dong, Makoto Onizuka, Chuan Xiao
Abstract
AI assistants store values that a language model derived from a person's records under the person's own definitions, such as their important contacts. Keeping such a value correct takes two steps: detection, noticing that an input changed, and repair, producing the correct new value. Benchmarks that score only the final answer cannot tell which step failed, and they state every change as new text. Seeing this limitation, we present MemShift, a benchmark of five shifts: source revision, deletion, redefinition, writer switch, and export. The first three fork a synthetic world into branches that differ only in the shift. Gold values and dependencies are fixed before any text, and a deterministic scorer replaces an LLM judge. For a memory that reports its invalidations, MemShift shows for each changed item whether the memory noticed the change and whether its repair was right. Moreover, we build a reference memory with known detection and repair to validate the measure. By evaluating four open-weight LLMs, we find that detection helps only as much as repair allows: the LLMs re-derive few values computed by the person's definitions correctly, and programs that the LLM writes over typed records close most of the gap to exact recomputation. Code and data are available at https://github.com/zichuanxu/memshift.