Memory systems struggle to remember user interaction preferences accurately
Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences
Artificial IntelligenceComputation and Language
Summary
Personal AI assistants can become annoying when they ask repeated questions or confirm things unnecessarily. This paper introduces MemPABench, a testing system that checks if AI agents remember how users want to be helped. The authors tested different memory systems with simulated users and found that current methods often fail to recall user preferences or adapt when those preferences change. Even when the right preference is known, agents often default to their usual way of acting.
What this means in practice
- •For personal assistant developers: Improve AI assistants by focusing on better memory systems that adapt to changing user interaction preferences over long periods.
- •For user experience designers: Design conversational agents that reduce repeated questions and unwanted confirmations by testing memory impact using MemPABench.
Authors
Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam
Abstract
Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.