Vision language teacher improves visual document retrieval accuracy

What Transfers from a VLM Teacher? Comparing Supervision Signals for Visual Document Retrieval

Information Retrieval

Summary

Finding the right document page for a query is hard when only limited correct answers are known. The authors show that using a vision-language model (VLM) teacher to better judge which pages are truly relevant helps retrievers learn more accurately, especially by identifying tricky false negatives. They compare different ways the teacher’s signals can guide learning and find that judging negative candidates is more effective than enriching positive examples. The teacher’s judgments align well with human annotations, especially when there is a clear single answer, improving retrieval performance notably.

What this means in practice

  • For search engine developers: Improve document search results by integrating teacher-guided judgments to better distinguish relevant from misleading pages during training.
  • For digital library engineers: Enhance retrieval accuracy in large visual document collections by using vision-language model signals to refine training negatives.

Authors

Saba Sturua, Han Xiao

Abstract

Visual document retrievers are trained contrastively: each query is matched to one page labelled relevant - the positive - and pushed away from negatives, pages presumed irrelevant. Recent methods distil a vision-language model (VLM) teacher into the retriever by enriching that positive, transferring the teacher's attention over it or a description of it. We ask whether the teacher is better spent on the other side, judging the candidates the retriever mines as negatives, which the label says nothing about. With student, data, optimizer and evaluation fixed, teacher-judged hard negatives and score distillation raise ViDoRe v2 nDCG@5 from 55.2 to 62.6 and 63.0; description alignment, as adapted here, gains 2.6 points and attention grounding nothing measurable. Against teacher-free rules that select four candidates from the same mined pool at identical training compute, the best of which is the positive-aware threshold current systems use, the teacher's judgement adds 4.1 points on v2 and 1.7 on v3. This is consistent with how incomplete the labels are. Annotators judge about two of a query's four top-ranked mined candidates relevant, none of them labelled, so training pushes the retriever away from relevant pages treated as negatives. What reaches the student is coarse: under a greedily decoded 0-100 rating prompt, 82% of the teacher's ratings come back at one end of the scale or the other, and a relevant/irrelevant partition keeps most of the distillation gain. A ten-annotator audit places the teacher within the range of variation among human annotators, and finds it reliable where a query has a single determinate answer. We release the code, the teacher's 3.3M judgements and page descriptions, the mined pools, the human audit and the trained adapters at https://github.com/elastic/vdr-teacher-signals.