Visual evidence grounding improves question answering in long documents
VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Computer Vision and Pattern RecognitionComputation and Language
Summary
Answering questions about long documents with lots of pictures, charts, and tables is hard because models need to find the right visual evidence first, then understand it. The authors created a big dataset called VELD-QA with 15,000 questions about real PDFs that have detailed markings showing which pages and visuals help answer each question. They tested various models and found that even when models know the right pages, understanding visual details and reasoning is still a big challenge. Adding detailed visual clues helps models get closer to the best possible performance, showing that such data is useful for improving these systems.
What this means in practice
- •For document ai developers: Improve systems that answer questions about complex documents by training on visual evidence annotated long documents.
- •For business intelligence teams: Enable better automated extraction and interpretation of data in reports through enhanced visual evidence grounding.
Authors
Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yazhe Hu, Lan Guo, Yuying Wang, Mengqing Guo, Bhavesh Dnyaneshwar Gore, Shubham Pathak, Ramapriya Ranganath, Sandeep Sharma, Tao Sheng, Sujith Ravi, Dan Roth
Abstract
Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.