HiVe-Mem improves multimodal agent memory with flexible evidence use
HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
When AI agents use many types of information like text and images over long conversations, they need better ways to remember and use past details. The authors created HiVe-Mem, which finds related memories by linking them in a graph and chooses the best parts of each memory to answer new questions. This method helps AI systems give better answers while using less computing power. The authors tested HiVe-Mem and found it worked well on several tasks that involve remembering and using multimodal data.
What this means in practice
- •For multimodal ai developers: Improve performance and efficiency of AI agents handling long-term multimodal interactions by integrating HiVe-Mem's memory expansion and evidence composition.
- •For conversational ai platform builders: Enhance chatbot memory systems to better retrieve and combine diverse past information for more accurate and context-aware responses.
Authors
Haozhen Zhang, Bohan Liu, Jianzhu Bao, Qingyuan Liu, Haodong Yue, Xiao Li, Zi Pong Lim, Wenya Wang
Abstract
As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \textbf{\hivemem}, a multimodal memory framework that interleaves \textbf{\textsc{H}}or\textbf{\textsc{i}}zontal Memory Expansion with \textbf{\textsc{Ve}}rtical Evidence Composition. For horizontal memory expansion, \hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that \hivemem consistently improves task performance across base models while reducing token consumption by about 90\% compared with the most efficient competing multimodal baselines.