Benchmarking multimodal retrieval systems on financial documents with complex visuals
VisR-RAG-Fin: Benchmarking Multimodal RAG over Visually Rich Financial Documents
Information Retrieval
Summary
Many financial reports use complicated layouts with tables, charts, and pictures that make it hard for computer systems to find and understand information. The authors studied how different computer methods that combine text and images can better find answers in these tricky documents. They created a new test called VisR-RAG-Fin that keeps the original look of these reports and measures how well various methods work. Their work helps developers know which techniques perform best when machines answer questions about visually rich financial papers.
What this means in practice
- •For enterprise software developers: Build systems that answer questions accurately from complex financial documents by selecting effective retrieval and generation methods tested on VisR-RAG-Fin.
- •For business intelligence teams: Evaluate multimodal tools that extract insights from visually rich financial reports to improve automated reporting and decision support.
Authors
Omar El Bachyr, Laura Maria Bernardy, Saad Ezzini, Jacques Klein, Tegawendé F. Bissyandé, Charles Mont'e, Anas Zilali, Ulrick Ble, Anne Goujon
Abstract
Enterprise documents such as financial reports are increasingly processed by automated systems, yet they are primarily designed for human reading and exhibit complex visual structure, combining dense text with tables, charts, diagrams, and rich layouts. These characteristics pose significant challenges for Retrieval-Augmented Generation (RAG) systems, which must retrieve relevant pages and leverage visual and structural information during answer generation. Despite recent advances in multimodal retrieval and vision-language models, there remains a limited understanding of how design choices across retrieval and generation components affect end-to-end RAG performance on visually rich documents. In this paper, we present a comprehensive empirical study of multimodal RAG for page-level Question Answering (QA) over financial documents. We introduce VisR-RAG-Fin (Visually Rich RAG for Finance), an open-source benchmark constructed through a general pipeline for visually rich documents that preserves original layout and visual content. Using this benchmark, we systematically analyze retrieval and generation behavior under text-only, multimodal, and hybrid configurations, providing a foundation for rigorous evaluation of RAG systems on complex enterprise document collections. Overall, this work provides actionable insights for practitioners designing and evaluating multimodal RAG pipelines over complex, visually rich enterprise documents.