H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions
2026-06-08 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created a new test called H2HMem to see how well language models remember things during complex conversations involving multiple people and different types of information, like text and visuals. They point out that current tests mostly focus on simple, one-on-one text chats, which don't reflect real-world conversations. Their study shows that current language models have trouble remembering and using information across different participants and formats. This means there's still a lot to improve for models helping in real-life group interactions.
large language modelsmultimodal memoryhuman-human interactionanaphoradeixisdyadic conversationsmulti-party conversationsmemory recallbenchmarklanguage understanding
Authors
Shiping Zhu, Yibo Yang, Zhengyang Wang, Tiancheng Shen, Dandan Guo, Ming-Hsuan Yang
Abstract
Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.