Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
2026-08-31 • Information Retrieval
Information RetrievalComputer Vision and Pattern Recognition
AI summaryⓘ
The authors point out that real-world information is often spread across many images and documents, making it hard for current models to answer questions accurately. To help with this, they created DocLongRAG, a big dataset with lots of questions linked to many images, which is closer to real situations. They also developed Doc-REFRAG, a system that smartly picks out relevant parts of images based on the question to speed up and improve answers. Their tests show that Doc-REFRAG works better and faster than other existing methods.
retrieval-augmented generationmultimodal documentsvisual tokensreinforcement learningdatasetquestion answeringinference latencyimage retrievalmachine learningDocLongRAG
Authors
Ruofan Hu, Shengyang Xu, Minjie Hong, Xiaoda Yang, Sashuai Zhou, Ke Lei, Tao Jin, Zhou Zhao
Abstract
Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question--answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.