MemoryAgentBench conflict scores largely match simple last-write rule
Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver
Artificial IntelligenceComputation and Language
Summary
This paper looks at how well a simple rule — always trusting the newest fact — does on a benchmark designed to test memory in AI models. The authors show that this rule alone solves over 80% of the questions, even without learning. They also find a specific set of challenging questions where the benchmark's answer can’t be reached by this rule, highlighting limitations in how facts are updated. Some advanced models do better on these tough cases, which are rare but important. The work helps clarify what the benchmark really measures and how memory conflicts are resolved in AI.
What this means in practice
- •For ai system developers: Improve fact conflict resolution in memory-based AI by comparing simple last-write rules to learned methods.
- •For knowledge engineers: Assess and debug benchmarks and datasets by identifying unreachable gold facts and understanding conflict outcomes.
Authors
Egor Pakhomov, Erik Nijkamp
Abstract
MemoryAgentBench's Conflict Resolution split is read as measuring "selective forgetting". We execute the benchmark's own rule - the newest statement about a fact wins - as a zero-learning resolver frozen on one of the four fact lists. Under the official metric the rule answers 80.25% of the questions (74.5% on the three held-out lists). Of the rest, 67 items have a released gold that the last-write graph cannot reach but overwritten statements would ("The capital of India is New Delhi." superseded by "The capital of India is Grosseto."; gold New Delhi); such items are a third of the multi-hop questions at 262K. Two long-context models and our pre-registered approximate re-implementation of the benchmark's BM25 agent, one retained run per item and outcomes only, score 84.7%, 82.6% and 41.6% on the items the rule solves against 10.4%, 11.9% and 6.0% on those 67. The failures are a reachability split plus a small parser-scope residual; the per-item split, not the aggregate, is the unit at which a score here can be read.