Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinement
2026-08-17 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors present Hyper-M2RAG, a new system for searching and generating information from documents that include different types of content like images, text, and tables. They use a special structure called a multimodal hypergraph to better capture complex connections among these parts, unlike traditional graphs that only link two things at a time. Their method also improves how documents split across pages are handled by focusing on important 'anchor' points, reducing effort and errors. Tests show their approach works better at finding and explaining information than previous methods.
Multimodal Retrieval-Augmented GenerationHypergraph Representation LearningN-ary relationshipsMultimodal HypergraphAnchor-driven Incremental RefinementCross-modal alignmentLong-form document processingRetrieval precisionGeneration coherence
Authors
Shenao Chen, Yidan Xu, Xiangmin Han, Rundong Xue, Duanpo Wu, Yuhan Gao, Chenggang Yan, Yue Gao
Abstract
Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data. Furthermore, existing refinement strategies often rely on exhaustive, full-page reconstruction to align cross-modal information, leading to prohibitive computational redundancy and the introduction of contextual noise in long-form document processing. In this paper, we propose Hyper-M2RAG, a novel framework that redefines multimodal document retrieval through High-order Hypergraph Representation Learning. We first formalize the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling. To mitigate semantic fragmentation caused by physical pagination, we introduce an Anchor-driven Incremental Refinement mechanism. Rather than performing a global sweep, our approach identifies boundary-crossing anchor nodes and reconstructs their local hyper-topology using one-hop neighborhood contexts. This targeted refinement effectively bridges cross-page knowledge gaps with minimal computational footprints. Extensive evaluations on multimodal benchmarking datasets demonstrate that Hyper-M2RAG significantly outperforms state-of-the-art methods in both retrieval precision and generation coherence. Our code is available at https://github.com/ShenAoChen2001/MMHRAG.