Benchmark reveals how AI agents remember working with users
DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users
Artificial Intelligence
Summary
Remembering facts about a user is not enough for AI agents that interact over a long time. The authors created DyadMem, a new test that checks if agents remember how to work well with individual users as their shared history grows. They show current AI models do well when given perfect memory but struggle when they must find and update memories themselves. This helps highlight where improvements are needed so AI can better assist users over many sessions.
What this means in practice
- •For chatbot developers: Evaluate and improve chatbot long-term memory by testing how well bots retain user-agent interactions over multiple sessions using DyadMem.
- •For ai product managers: Assess pipeline weaknesses in AI systems’ ability to remember task-related user interactions and update memories accurately for enhanced user experience.
Authors
Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang, Tengda Guo, Wentao Qiu, Ying Wang, Liujian Tang
Abstract
Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves. Existing benchmarks primarily supervise user facts and preferences or experience reusable across users, leaving this relationship-specific agent memory implicit. Additionally, most prior works measure the model solely with final-answer QA over long interaction histories, making the assessment still incomplete and unreliable. To this end, we introduce DyadMem with the proposed new definition User-conditioned Relational Agent Memory (URAM). DyadMem jointly annotates user-side memory and URAM along the same multi-session trajectories, resulting in 6 memory categories. To summarize, it includes 3,065 episodes, 50,961 sessions, and 61,210 QA instances, with extensive session-level Capture and Update gold annotations, query-level Recall support, and two QA settings: Gold-Memory and Full-Pipeline. Across 16 open-weight and 4 proprietary models, Gold-Memory QA is consistently strong, yet Full-Pipeline QA drops sharply. Such a gap explicitly supports our fine-grained evaluation design. Additionally, several quantitative results further reveal low Capture recall, incomplete Recall, and unsafe-deletion issues arising from even the frontier LLMs. We further conduct a rigorous experiment to validate the effectiveness of our URAM and observe the positive effects for all 20 models. In summary, DyadMem is a dual-domain, full-pipeline memory benchmark with extensive annotation efforts for advancing the domain's development.