Benchmark reveals failures in memory and reasoning for multi-modal assistants
MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents
Computation and LanguageArtificial Intelligence
Summary
Smart assistants need to remember not just words, but also images, videos, and sounds to understand you better over time. The authors introduce MOSAIC, a new test with many questions to check how well assistants remember and reason using all these different types of information across long conversations. They found that assistants often lose important details depending on how information is stored or retrieved and can struggle to know when to say "I don't know." This test helps pinpoint exactly where these memory and reasoning problems happen, instead of just giving a single score.
What this means in practice
- •For conversational ai developers: Evaluate multi-modal assistants’ memory by pinpointing where information is lost in their handling of text, images, audio, and video over long conversations.
- •For customer service chatbot teams: Improve chatbot memory and response reliability by diagnosing failure points in remembering and using diverse media during extended user interactions.
Authors
Hong Ting Tsang, Boyi Zhang, Haoyu Huang, Jiaxin Bai, Yufei Li, Yisen Gao, Zhongwei Xie, Hongyu Luo, Jeff Z. Pan, Yangqiu Song
Abstract
Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report a single end-to-end score, so they cannot locate whether evidence is lost when it is represented, retrieved, delivered, or read. We introduce MOSAIC, a benchmark of 923 questions over persona-grounded histories of 11-24 sessions spanning text, image, video, and audio. MOSAIC constructs targeted spatial, temporal, and prosodic signals rather than decorative media, with the strongest causal validation for spatial information and selector-level validation for annotated prosody, and pairs them with a matched diagnostic protocol that freezes the question, reader, prompt, and budget while intervening on representation, retrieval, and delivery. Matched decomposition shows that representation loss is modality-dependent, residual reading dominates even under gold evidence, and abstention remains weak on these controls: strong answerable scores coexist with failures to refuse. MOSAIC thus turns omni-modal memory evaluation from a leaderboard into a diagnosis of write, retrieve, deliver, and read failures.