Agent memory systems struggle to learn user habits from weak clues
HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory
Artificial Intelligence
Summary
When people interact over time with AI assistants, they often reveal habits by their choices. This paper shows that just remembering past conversations doesn’t help AI understand these habits well, especially when clues are unclear or habits change. The authors created a test called HABIT-Bench to see how well AI memory methods can guess and use habits during long interactions. They found that current methods often perform poorly, even with clear hints, highlighting a big challenge in making AI better at personal memory.
What this means in practice
- •For dialogue system developers: Improve AI assistants by testing memory methods to better recognize user habits from partial and changing clues.
- •For customer service platform engineers: Integrate benchmarks to assess whether virtual agents can learn ongoing customer preferences despite inconsistent feedback.
Authors
Junming Zhang, Rubin Wei, Xu Cheng, Hao Sun, Ruoxi Cheng, Xiquan Wang, Peisong Wu, Mingsong Chen, Zhouhan Lin
Abstract
In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies. Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately. We introduce HABIT-Bench to evaluate how well agent memory systems infer habits from weak evidence and apply them to later requests. We construct synthetic user histories from task-oriented dialogue data and controlled habit graphs. The histories span hundreds of sessions across food, finance, software, and travel. Four-choice tasks assess agents' ability to infer user habits and apply them appropriately. They include temporary exceptions, changes in habits, and suggestions that users have not accepted. Sampled tasks undergo independent human review. We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieves 33.61\% macro accuracy across the four domains. Performance remains limited even when annotated evidence is supplied directly. These results expose a persistent gap between access to past interactions and the ability to infer and appropriately use user habits.