Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and LanguageInformation Retrieval
AI summaryⓘ
The authors studied how models can better show where their answers come from in visual documents. They found that models using text quotes as evidence were much better at pointing to the right parts than models that only gave box coordinates. They also developed a way to train models to improve this quoting ability without needing expensive detailed labels. This method improved the accuracy of models in attributing their answers to the correct evidence. Overall, the work suggests a better way to link answers to evidence using language and retrieval instead of just coordinates.
Visual document understandingAttribution hallucinationCoordinate interfaceVision-language modelsBilingual CiteVQAEvidence recallMultimodal retrieverLayout parserReinforcement learningRegion-level supervision
Authors
Zhuchenyang Liu, Yao Zhang, Yu Xiao
Abstract
Reliable visual document understanding requires a model to attribute each answer to the evidence regions that support it. Recent benchmarks and systems express this step through a coordinate interface: the model outputs the coordinates of bounding boxes that mark the evidence regions in the document. Under this interface, vision-language models often fail to identify the right regions even when the answer is correct, a failure known as Attribution Hallucination. We present a study that investigates whether this failure is partially limited by what the model can express through coordinates. On a verified bilingual CiteVQA subset, we compare the coordinate interface with a language interface in which the model outputs only text, quoting its evidence verbatim, and a multimodal retriever returns the location of each quote as a page region proposed by a layout parser (tables and figures are quoted through their captions or notes); the comparison is repeated over six open vision-language models. Compared with the coordinate interface, evidence recall rises from at most 8 points to between 26 and 47 and the hallucination rate roughly halves, with little change in answer quality. Building on this comparison, we use the same quote-and-retrieve pipeline as a training scaffold: because region-level evidence labels are expensive to collect for long documents, we introduce a GRPO recipe whose reward is a judge's reading of the gold answer and crops of the retrieved regions, training the model to quote better evidence without any region labels and raising an 8B backbone's strict attributed accuracy from 22.4 to 33.8. These findings indicate a practical path to improve attribution"without a coordinate interface and without costly region-level supervision.