DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
2026-06-22 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created DynamicMem, a new benchmark to test how well AI assistants can remember and update detailed user profiles over a long time (15 months), across many apps and evolving behaviors. They highlight that real user memory is complex because profiles change differently over time and clues are scattered, unlike simpler previous tests. Their results show current systems struggle to maintain accurate memories as history grows and often fail because the memory retrieval, not the AI's answer writing, is the main problem. This means future work should focus on improving memory access rather than just AI generation.
LLM agentsuser profile memorylong-term user modelingsynthetic benchmarkmemory retrievalmulti-app dataattribute evolutionpreference inferenceprofile reconstructionAI personal assistants
Authors
Wenya Xie, Shengming Zhou, Zelin Li, Pouya Parsa, Shuang Zhou, Xinheng Ding, Chinmay Arvind, Guanchu Wang, Vladimir Braverman, Ali Payani, Yantao Zheng, Zirui Liu
Abstract
LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/