Evaluating false memory in AI agents using counterfactual reasoning
Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning
Artificial IntelligenceMachine Learning
Summary
AI agents use memory to solve problems beyond what they have directly experienced, but sometimes their memories can be wrong or misleading, called false memory. The authors introduce a tool called FAME that checks for these incorrect memories by imagining 'what if' scenarios to see how the AI’s beliefs would change. This approach helps detect false memory better than just looking at the AI’s answers. They tested it on tasks like math, coding, and complex reasoning and found it works well across different situations.
What this means in practice
- •For ai system developers: Detect and reduce false memory errors in AI agents to improve their performance in new or shifted environments.
- •For software testing teams: Use counterfactual templates to evaluate AI-generated code for hidden biases or incorrect assumptions before deployment.
Authors
Quan M. Tran, Zhuo Huang, Zhen Fang, Jing Zhang, Mingming Gong, Tongliang Liu
Abstract
Autonomous agents increasingly rely on memory to generalize beyond their training environments. However, agents are bounded by what they have seen and believed, and leveraging such memories in unseen environments can introduce biases into their internal beliefs. We formalize this phenomenon as \textit{false memory}, which can arise from spurious correlations, environment shifts, and knowledge conflicts. Despite its importance, false memory is difficult to evaluate because it stems from agent internal beliefs and is easily confounded with ordinary generalization failures. Therefore, we propose FAME, a training-free framework that evaluates false memory through the evolution of agent beliefs under counterfactual reasoning. Specifically, counterfactual scenarios reveal how beliefs change as the latent concept of memory shifts under hypothetical interventions; thus, measuring the resulting concept drift provides a signal for distinguishing faithful versus false memory. Such concepts can be estimated from agent hidden states before answer generation, avoiding the need for reward design or answer sampling. Empirical experiments reveal that simply monitoring answers often fails to detect false memory, while FAME achieves AUROCs of 76.2% - 96.7% across false-memory settings, and outperforms the best baseline by 3.4% - 23.3% across realistic benchmarks, spanning math reasoning (GSM-Symbolic), code generation (GitChameleon), and complex reasoning (BigBench-Hard). We further release corresponding counterfactual templates and facilitate future research on false memory.