Document layout helps pinpoint answers in visual question answering tasks
DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering
Computer Vision and Pattern Recognition
Summary
Finding the exact part of a document that answers a question is tricky because most tests don't show which elements support the answers. The authors created a big dataset called DocAttriBench that links answers to parts like tables or text blocks. They used a clever method called MAPPET, hiding parts of the document to see which part makes the answer easiest to guess. They tested many language models and found even big ones often miss the correct source inside the document. This new dataset helps make question-answering tools for documents more accurate and reliable.
What this means in practice
- •For document processing teams: Develop visual question answering systems that can pinpoint exact document parts backing their answers for more trustworthy outputs.
- •For business intelligence analysts: Use grounded document Q&A to verify and trace data sources in reports or presentations automatically.
Authors
Luca De Grandis, Silvia Cappelletti, William Raccagni, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Abstract
Answer grounding in document visual question answering remains an open challenge: most benchmarks lack grounding annotations or provide limited-quality labels, while constructing grounded datasets still requires costly manual effort. We introduce DocAttriBench (DAB), a large-scale benchmark for fine-grained, element-level source attribution in Document VQA, grounding answers to specific layout elements such as text blocks, tables, and images. To build DAB, we propose a Mask-based Perplexity-Derived Attribution method (MAPPET) that combines document layout and language modeling to identify the most informative element for each answer. MAPPET measures the increase in perplexity after masking candidate elements and attributes the answer to the element contributing most to model confidence. Applying MAPPET to multiple existing Document VQA datasets yields DAB, with 237k documents and 296k question-answer pairs with element-level grounding. We benchmark grounding-capable multimodal LLMs on DAB, evaluating answer accuracy, attribution accuracy, and overall answer quality. Results show that while larger models generally achieve higher answer accuracy, even the strongest models often fail to localize the supporting elements. DAB provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models. Dataset and code are available at https://aimagelab.github.io/DocAttriBench/.