Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation
Machine Learning
Summary
The authors looked at how AI systems handle information that updates over time, like threads or histories, where some facts become outdated or replaced. They tested different ways of keeping track of these changes, including simple lists, marking outdated info broadly, and a detailed system called RevisionLedger. Their study found that the detailed system didn't improve accuracy once they accounted for how results were shown to judges. Instead, a simpler method that just marks old info as invalid worked best for current questions. The authors suggest that evaluations should keep the display format consistent and that simpler methods might be enough for tracking changes in memory.
Authors
Zhaoyang Jiang, Zhizhong Fu, Zicheng Li, Yunsoo Kim, Jiacong Mi, Xuanqi Peng, Fei Teng, Honghan Wu
Abstract
AI systems increasingly retrieve from records that revise themselves: issue threads, encyclopedic histories, policy logs, and long conversations. The challenge is not only finding relevant evidence, but deciding which claims remain in force, which were superseded, and when to abstain. Structured memories promise to solve this with typed edges, temporal updates, and conflict status, yet evaluations often change mechanism and prompt presentation together. We study this as Evidence-State Revision, comparing flat retrieval, coarse edge invalidation, and fine-grained RevisionLedger on 2,907 high-agreement questions from GitHub, multi-repo issue histories, Wikipedia, and DyKnow-style temporal streams. A render-matched control (same layout, deprecation disabled) reveals the central confound: when a value is changed and later restored, RevisionLedger appears to beat a flat baseline by +0.182, but almost all the gain comes from easier presentation; the fine-grained mechanism residual is indistinguishable from zero (+0.021 to +0.025 across two judge families). After presentation is controlled, coarse invalidation is the only mechanism that pays for current-state queries, beating the fine ledger by 0.084; the same query-sufficiency principle says provenance mainly needs retained invalidated evidence, not richer typing. Memory evaluations should hold render fixed, and deprecation-aware systems should deploy the coarsest retained state that covers their queries.