Benchmark tests memory of subtle cues in multimodal conversations
CUE-Mem: Benchmarking Long-Term User Memory via Implicit Cues in Multimodal Conversations
Artificial IntelligenceComputation and Language
Summary
Remembering details from long conversations where users share images and sounds is tricky because some memories come from subtle hints, not clear statements. The authors created a new test called CUE-Mem that checks how well systems can recall these hidden clues across images, sounds, and text during long chats. They found computers struggle most with spotting and using these quiet hints, especially when relying only on text descriptions instead of directly accessing images and sounds. Even giving more detailed text helps a bit but gets expensive, and having direct access to multimedia doesn't always solve the problem.
What this means in practice
- •For conversational ai developers: Test and improve dialogue agents that remember users’ subtle background details without explicit statements by using the CUE-Mem benchmark.
- •For customer support platform teams: Enhance support bots to better reference implied context from customer images and sounds over long conversations.
Authors
Yulin Hu, Yanyan Zhao, Zimo Long, Xing Fu, Mengtong Ji, Weixiang Zhao, Yutai Hou, Qianchao Wang, Dandan Tu
Abstract
Long-term memory is essential for multimodal agents that interact with users across sustained conversations. However, user memories are not always explicitly stated: they may also be implied by recurring background objects in images, ambient sounds in audio, or other peripheral multimodal cues. Existing benchmarks largely focus on text-only memory or explicit multimodal evidence, leaving implicit multimodal cues underexplored. We introduce CUE-Mem, a text-image-audio benchmark for evaluating long-term user memory from implicit cues. CUE-Mem contains 2,674 questions across explicit and implicit evidence settings and covers four tasks: Entity Recall, Long Pattern, Personalized Recommendation, and Answer Refusal. Across textualized memory systems, implicit performance remains far below oracle evidence, locating the main bottleneck in preserving and retrieving subtle cues rather than question answerability. Increasing caption detail recovers more of this evidence, but brings uneven gains and rapidly growing token costs, motivating native multimodal access. Yet native access does not uniformly resolve the bottleneck: evidence use depends strongly on the backbone, while multimodal indexing introduces substantial retrieval noise. CUE-Mem provides a testbed for memory systems that selectively retain, retrieve, and use subtle multimodal evidence.