Summary
Answering complicated questions from documents that have lots of text, tables, pictures, and captions is hard because the information is spread across different parts and types. The authors created a new test called TOPOGRAPHRAG-BENCH to check how well computer systems can connect these different pieces of information visually and textually to find the right answers. They tested several systems and found that those using both images and text work best but still struggle to fully understand how different parts relate. Systems that only use text or only look at whole pages don't do as well. This shows that future methods need to better understand how document layouts and different types of information fit together.
What this means in practice
- •For document ai developers: Build systems that answer complex queries by connecting text, tables, and figures within document layouts for better multimodal reasoning.
- •For enterprise knowledge management teams: Improve document search tools by integrating layout-aware multimodal evidence to recover how information units relate across pages.
Authors
Ruochi Li, Jianzhe Lin, Haoxuan Zhang, Haihua Chen, Junhua Ding, Edward Gehringer, Yang Zhang
Abstract
Real-world documents distribute evidence across text, tables, figures, and captions within complex page layouts. Answering complex questions over such documents therefore requires more than retrieving relevant passages: systems must recover the evidence topology that connects heterogeneous evidence units. Existing GraphRAG evaluations remain largely text-centered, while multimodal document RAG benchmarks assess cross-modal retrieval and generation without directly evaluating recovery of the intended evidence topology. We introduce TOPOGRAPHRAG-BENCH, a layout-grounded benchmark for multimodal evidence reasoning in GraphRAG, comprising 2,024 questions over 201 long, visually rich documents. Questions are constructed bottom-up from text, figure, and table evidence units under three controlled topologies: single-hop retrieval, bridge-chain reasoning, and multi-source synthesis. To ensure that questions preserve their intended structure, we apply counterfactual validation for shortcut resistance, modality necessity, and evidence necessity. We evaluate text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems using retrieval, generation, and topology-aware reasoning metrics. Multimodal GraphRAG systems achieve the strongest overall performance, but still fail when visual-textual evidence alignment or multi-unit composition is incomplete. Text-only GraphRAG struggles when key dependencies are grounded in figures or tables, while page-level visual retrieval lacks the fine-grained structure needed for topology recovery. These findings motivate GraphRAG systems that move beyond text-derived entity relation graphs to explicitly model document layouts, cross-modal evidence alignment, and the reasoning roles of evidence units. Code and data are available at https://richardlrc.github.io/TopoGraphRAG-Bench/.