Hypergraph method improves evidence selection for long multimodal documents
Hypergraph-Enhanced Evidence Selection for Long-Context Multimodal Document QA
Information RetrievalDatabases
Summary
Answering questions from long documents that contain both text and images is hard because important clues are scattered and systems have limited space to process them. The authors propose a new method called HyperDoc that organizes document parts into a network showing their relationships and picks the most helpful parts based on the question and how much information can fit. This approach helps keep important connections and reduces information loss. Tests show that HyperDoc works better than previous methods on several datasets.
What this means in practice
- •For document ai developers: Select the most relevant text and images from long documents for question answering within computational limits using the hypergraph-based method.
- •For business process automation teams: Improve automated extraction of key information from multi-page reports that include tables and images by preserving document structure during retrieval.
Authors
Wenxin Zhao, Jiachuan Wang, Xinyi Zhu, Jiake Ge, Haoyang Li, Yongqi Zhang, Shuangyin Li, Lei Chen
Abstract
Long-context multimodal document question answering (QA) has become an important yet challenging task for modern vision-language models (VLMs), as it requires reasoning over evidence dispersed across pages and modalities within limited input budgets. Current pipelines adopt retrieval-augmented generation (RAG) to select multimodal evidence for downstream VLMs. However, existing systems mainly rely on static selection rules over fixed-granularity retrieval units such as pages or blocks, resulting in information loss during retrieval and suboptimal performance in downstream reasoning. To alleviate these issues, we formulate the core task as budget-aware evidence selection over structured document evidence and propose HyperDoc, a document RAG framework that constructs compact yet sufficient evidence while preserving document relations. Specifically, HyperDoc constructs a document hypergraph to explicitly capture structural, logical, and cross-modal relationships among multimodal document elements. Over this structured evidence space, HyperDoc performs adaptive cost-aware hyperedge selection based on query intent and input budget. Extensive experiments on three representative benchmarks demonstrate that HyperDoc outperforms various types of state-of-the-art baselines.