Generative feedback improves region retrieval in visual documents
GARRET: Generative Feedback for Region Retrieval in Visual Documents
Computer Vision and Pattern RecognitionInformation Retrieval
Summary
Finding specific parts of complex documents like forms or articles can be hard when systems only look at whole pages. The authors introduce GARRET, a way to train a system to find smaller regions of a page that answer a question, using only examples of whole pages with questions and answers, without needing detailed labels. The system uses a special learning step where it softly decides which parts of the page to focus on and gets better at picking important areas over time. This method works better than previous tools and uses fewer resources.
What this means in practice
- •For document processing teams: Improve retrieval of specific parts of scanned or digital documents to answer questions more accurately without needing detailed annotations.
- •For digital archiving teams: Enable fine-grained search within large document collections by indexing and retrieving relevant regions rather than full pages, reducing storage and computation needs.
Authors
Weijian Jian, Siyang Chen, Yuguang Yang, Yajun Zhang, Chunyu Xie, Dawei Leng, Yuhui Yin
Abstract
Multimodal retrieval-augmented generation over visually rich documents requires evidence at a finer granularity than entire pages. Existing visual retrievers typically index or return full pages, whereas training region-level retrievers often requires costly query-to-region annotations or noisy model-generated labels. We introduce GARRET, a region-retrieval framework trained from (page, question, answer) triplets through differentiable generative feedback. A shared context-preserving retriever produces page and region multi-vector representations. During training, their retrieval scores define a soft routing distribution over candidate region crops. An auxiliary vision-language model attends to each crop separately, aggregates crop-specific contexts using the routing weights, and predicts the question followed by the answer under a next-token prediction objective. Gradients through the routing distribution assign query-conditioned credit to the indexed region representations, without requiring query-to-region annotations. At inference, the auxiliary model is removed. GARRET also reuses the contextualized region representations, together with page summary tokens, to construct a compact page representation. We introduce ViDoRe-V3-Region, a corpus-wide benchmark containing 200,682 fixed candidate regions across eight domains. GARRET achieves a macro nDCG@10 of 0.4126, outperforming the strongest visual baseline by 2.94 points. On page retrieval, it remains comparable to ColQwen2.5-v0.2 while indexing 47.8% fewer vectors per page.