Multimodal agents remember entities and changes using hypergraph memory
Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents
Computation and Language
Summary
Remembering detailed visual information and how things change over time is hard for AI agents that process images and text. The authors propose a new memory system called MHG-Mem that keeps track of individual objects, their facts, and how they change, all linked to the original evidence. This system organizes memories in a connected structure called a hypergraph and allows the agent to carefully check details or understand changes across different sessions. Their tests show this approach helps the agent remember and reason better compared to previous methods.
What this means in practice
- •For robotics developers: Enable robots to remember objects and track their changes across multiple interactions by linking observations accurately.
- •For interactive ai system engineers: Improve AI assistants with detailed memory that verifies visual details and tracks evolving information over time for better context-aware responses.
Authors
Jielong Tang, Ling-An Zeng, Xujie Yuan, Xiao Dong, Zhiqian Xia, Qingfeng Li, Shixiong Kai, Kaichao Liang, Rong Yao, Jian Yin
Abstract
Long-term memory for multimodal agents remains challenging, as it must preserve fine-grained visual evidence while tracking entity evolution across sessions. Existing abstraction-based memories compress images into text, often discarding visually discriminative details, whereas pixel-preserving memories retain raw observations but struggle to reliably associate diverse observations with persistent entity identities and track their evolving states. To address these limitations, we propose MHG-Mem, a multimodal hypergraph memory that jointly organizes entity identity, state evolution, and provenance to visual evidence. Specifically, MHG-Mem represents distinct entity mentions and their observed facts as individual nodes, grouping identity-consistent mentions and facts through occurrence hyperedges. It then synthesizes aspect-specific evolution states from these grouped observations and links them to supporting entities, facts, and source evidence through evolutionary and evidential hyperedges. To navigate this structured memory, we introduce hypergraph-based agentic search, which adaptively selects an entry layer, traverses typed hyperedges across memory levels, and inspects source observations when fine-grained verification is required. These designs support both detailed visual verification and reasoning over cross-session state changes. Extensive experiments across three benchmarks demonstrate the effectiveness of MHG-Mem, improving performance by 8.20-9.98% over the strongest baseline.