Personalized assistants struggle to give good advice in long chats
PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations
Artificial Intelligence
Summary
When people use AI assistants over a long time, these assistants find it hard to remember and use the right parts of their past talks. The authors built a test called PRAGMA to see how well different systems can guide users by using their conversation history. They found existing systems often fail to pick the best memory bits and use them correctly for personalized advice. This shows we need better ways for AI to remember and reason over long interactions.
What this means in practice
- •For chatbot developers: Improve chatbot systems that provide personalized advice by testing memory retrieval and reasoning based on users' long-term interactions.
- •For customer support teams: Develop better support tools that consistently use past customer conversations to tailor recommendations and solutions.
Authors
Hyojeong Yu, Hyukhun Koh, Minsung Kim, Yunah Jang, Kyomin Jung
Abstract
Large language models (LLMs) are increasingly deployed as personalized assistants that interact with users over extended periods of time. As conversations grow longer, relying on full interaction histories becomes increasingly inefficient and unreliable: long contexts introduce substantial computational overhead, making it difficult for models to consistently identify and utilize the most relevant information for the current request. These challenges have motivated memory systems that structure and retrieve user-specific information. In realistic interactions, users often seek practical guidance such as recommendations, planning, and decision support. Unlike factual recall tasks, personalized guidance requires models to integrate information across multiple past conversations and reason about changing user preferences and experiences. However, existing conversational memory evaluations mainly focus on retrieval and factual recall. To study this challenge, we introduce PRAGMA, a benchmark for evaluating personalized guidance in long-term conversations. PRGAMA contains curated longitudinal conversation histories, evidence annotations, and guidance scenarios grounded in evolving user contexts and incorrect user assumptions. Experiments across retrieval systems, memory systems, and long-context models reveal that current systems struggle both to recover the appropriate conversational evidence and to effectively use it for personalized guidance. Our results highlight the need for memory architectures that support robust conversational retrieval and memory-grounded reasoning beyond evidence recall.