Visprove improves detailed visual evidence attribution in scientific qa
VisProve: Learning Typed Visual Provenance for Scientific Document Question Answering
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Answering questions on scientific documents is tricky because it's important to know not just the answer but how the sources support it. The authors introduce a way to label parts of documents to show if the information is directly quoted, summarized, or inferred. They created a dataset called ReViE and built a tool named VisProve that improves the ability to link answers to specific visual parts of documents and explain their support type. While VisProve shows better pinpointing of supporting evidence, accurately matching answers with detailed evidence remains a challenge.
What this means in practice
- •For document ai engineers: Enhance scientific QA systems by linking answers to precisely typed visual evidence within full documents.
- •For legal document analysts: Use typed visual provenance to clarify how textual and visual content supports legal query answers.
Authors
Jingxuan Wei, Zhengyang Wang, Zheng Guo, Xiaoming Wu, Fuqiang Wang, Xin Wang, Bihui Yu, Yining Wang, Junnan Zhu
Abstract
Visual citations show where an answer is supported, but not how the cited content contributes to it. In scientific document question answering, a model may read a value directly, compress several observations, or combine evidence to infer a conclusion. We study typed visual provenance, where a system generates an answer together with source regions and explicit support relations. We introduce ReViE, a dataset of 3,941 questions over complete scientific documents, with page, section, and region annotations labeled as Quotation, Compression, or Inference. We further propose VisProve, which augments joint answer and provenance generation with differentiable Spatial Evidence Reading and Evidence-Answer Relation Feedback. VisProve conditions next-token prediction on localized visual evidence and its estimated support relation through a gated norm-bounded residual. On the document-disjoint ReViE test set, VisProve improves Typed-F1 from 3.52 to 5.48 and Section-F1 from 45.99 to 53.24 over joint supervised fine-tuning with the same backbone, while Page-F1 changes only marginally. These results suggest that coupling visual grounding with support relation modeling mainly improves fine-grained provenance, while the low absolute Typed-F1 reveals a substantial gap between answer generation and precise visual evidence attribution.