Memory use in language models does not always improve user preferences
Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
Computation and LanguageArtificial Intelligence
Summary
Language models can remember information about users to make conversations more continuous and personalized. However, the researchers found that having memory does not always lead to responses users like better; sometimes users prefer answers without memory. They also showed that current automatic systems that rate responses often disagree with what people prefer. To fix this, the authors created a model that learns from user feedback to predict when memory use is actually beneficial.
What this means in practice
- •For chatbot developers: Design chatbots that decide when to use stored user information based on predicted user preferences, improving response satisfaction.
- •For customer service teams: Improve automated support by integrating memory-aware systems that tailor responses to what users actually prefer, not just what’s stored.
Authors
Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu, Michael J Ryan, Jan Batzner, Zheng Hui, Shaolong Wu, Orestis Papakyriakopoulos, Alex Pentland, Nigel Collier, Jiaxin Pei
Abstract
Memory enables language model assistants to maintain continuity across sessions and generate personalized responses, but knowing more about a user does not necessarily produce responses they prefer. Existing evaluations assess memory capabilities without directly establishing whether memory use benefits the user. To study this gap, we collect approximately 1,000 pairwise judgments from 87 participants using their own ChatGPT queries and personal memories. We find that memory-conditioned responses are not consistently preferred over responses generated without memory and in some cases, users strongly prefer the latter. Automatic evaluators also show limited agreement with users and favor memory-conditioned responses substantially more often than users do. Motivated by these gaps, we introduce a memory-aware reward model that learns from user judgments, achieving 72.8\% preference prediction accuracy and outperforming the existing baselines. Our findings motivate evaluation and optimization objectives that align memory use with the preferences of the people whose information is being used.