Redundant facts in long documents challenge standard citation scoring
One Fact, Many Places: Redundant Evidence in Long Visual Documents Breaks Single-Gold Evaluation of Retrieval, Citation and Attribution
Computer Vision and Pattern Recognition
Summary
When computers try to find where a fact comes from in long documents with text and pictures, they usually expect that each fact appears in only one place. But in reality, the same fact can appear multiple times, such as in charts and text. The authors showed that this redundancy happens often and causes current systems to be unfairly penalized for finding correct facts in different places. They created new annotations and scoring methods that account for these repeated facts, improving how we evaluate fact retrieval and citation.
What this means in practice
- •For document search engineers: Improve search tools for long visual documents by accounting for multiple fact locations, leading to better evaluation of retrieved evidence.
- •For digital archive managers: Enhance document indexing to recognize redundant facts across text and images, improving retrieval accuracy in large collections.
Authors
Xiaokang Yuan, Binhui Yao, Luwei Yang, Ruimin Chen, Zeng Wang
Abstract
Document understanding is increasingly judged on where an answer comes from, scored against annotated gold locations as if each fact lived in one place. In long visual documents, a fact often lives in several places: a chart value is restated in the text, a key figure reappears in the summary. We ask how often such copies occur in standard benchmarks and what they do to evaluation. We annotate 420 questions from MMLongBench-Doc and MMDocRAG with the locations that independently establish each required fact, checked by two independent reviewers. Copies are common and often cross-modal: 19.5% of MMLongBench-Doc test questions have a strict copy outside the gold, and in 28% of MMDocRAG questions a non-gold candidate alone supplies a required fact. Single-gold scoring then fails in three predictable ways. A retriever that finds a copy is scored as a miss: page recall@1 of text and visual retrievers is underestimated by 13-24 points on questions with a cross-page copy. A model that cites a copy is scored as citing wrongly: crediting equivalent evidence reverses 11% of pairwise orderings among 65 competitive MMDocRAG systems. Masking the gold region leaves the fact available elsewhere: with a copy in the input, 41-56% of correct answers across four models survive it yet fail once every annotated instance is masked. Set-level masking with automatically found copies closes 78% of the gap to masking every annotated instance at matched granularity (three models); an unverified key-string rule lowers survival at least as far but masks three times the area. We release the annotations and redundancy-aware metrics.