User simulators show human-like memory effects but forget facts differently
Analyzing Repetition and Interference Effects in Memory-Constrained User Simulators
Computation and LanguageArtificial Intelligence
Summary
Testing how computer programs mimic human memory, the authors looked at whether these programs respond like people do when facts are repeated or mixed with confusing information. They found some programs match overall human accuracy and behave similarly with repetition and interference, but they make mistakes in different ways—humans misremember, while programs forget. The authors provide a set of test questions to help future work build better user simulators.
What this means in practice
- •For conversational ai developers: Improve user simulators to more accurately reflect how humans remember and forget information during interactions.
- •For cognitive computing teams: Use the benchmark data to test and refine memory models in systems that simulate human-like information processing.
Authors
Cara Su-Yi Leong, Nicholas Tomlin, Tal Linzen
Abstract
User simulators can help evaluate and train AI systems to interact with humans at scale. To do so, evaluation of user simulators must measure behavioral parity with people. We evaluate whether memory-constrained user simulators are sensitive to two factors known to affect human information retention: repetition of a fact and interference from competing information. We find that some simulators achieve human-like overall accuracy, and replicate the magnitude of repetition and interference effects, but none match people's accuracy on particular questions: while humans misremember facts, models forget them, leading to different error patterns. We release this data as a challenge benchmark for future work on user simulators.