Benchmarking ai companions understanding through real months long chats
RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations
Artificial Intelligence
Summary
This paper looks at how well AI companions can understand people based on conversations that last months. The authors collected real chat data from ten relationships, with over 27,000 messages, to see when and how memory of past chat helps AI. They found that most of the time, AI can respond well using recent messages, not needing to remember far back. It’s also hard for current methods to tell when remembering past messages is actually needed. Finally, different AI systems were similarly good at figuring out a person’s profile, even when their computing costs varied a lot.
What this means in practice
- •For chatbot developers: Use benchmark data with real multi-month conversations to improve AI memory and persona understanding models in companion systems.
- •For customer service teams: Test dialogue systems on long interaction records to better handle when historical context matters for customer inquiries.
Authors
Arman Behnam, Sunglyoung Kim, Liangwei Yang
Abstract
A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing this requires a real person's record, and such records are private, so benchmarks generate the person and the questions and settle in advance what matters. We release \bench, ten real relationships with an AI companion: 27,218 messages over up to 120 days, released as the conversation and four files derived from it, a profile, a persona, a chat ground truth and a question set, each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation. Three findings follow. First, the past is rarely needed and far away. Pooled measures mislead: a recency window finds the required message for 95.9\% of probes and 2.2\% of those that need memory, and at the natural rate 96\% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector we tried can tell when memory is needed on real messages, authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost.