Visual document retrieval improves by adapting queries with residual feedback
Test-Time Adaptation with Query-Dependent Residuals for Visual Document Retrieval
Information Retrieval
Summary
Visual document retrieval systems find relevant pages in large document collections using precomputed page data, but adapting them after deployment is hard. The authors present Q-REACT, a method that uses feedback from a secondary ranking process to adjust queries without changing the underlying document data. This approach allows all documents to compete fairly during retrieval and works well across multiple tasks and retrieval systems. It improves performance with minimal extra computing time and does not require changing or re-encoding the documents.
What this means in practice
- •For enterprise search teams: Improve document retrieval performance on existing indexes without re-encoding documents or retraining encoders, using limited feedback from rerankers at query time.
- •For digital archive managers: Enhance relevance of search queries on visual document collections by adapting queries on the fly without rebuilding the archive embedding index.
Authors
Zeliang Li, Xiaofen Xing, Kailing Guo, Xiangmin Xu
Abstract
Visual document retrieval (VDR) systems depend on page embeddings computed before deployment, which makes adaptation difficult when encoder parameters or corpus re-encoding are unavailable. Rerankers provide useful relevance signals, but conventional reranking applies them only to selected queries and candidate pages. We introduce Q-REACT, a query-side test-time adaptation method that converts limited reranker feedback into reusable retrieval improvements. Q-REACT learns a shared low-rank transformation that produces query-dependent residuals, combines adapted query scores with document-level context, and distills reranker preferences with a student distribution normalized over the complete task-specific page index. This design lets unscored pages compete through cached embeddings while keeping the encoders and page index fixed. Across eight ViDoRe V3 tasks and five open-weight and proprietary backbones, Q-REACT improves average retrieval over evaluated baselines at sparse and full-coverage budgets, transfers to held-out queries and tasks, and adds little inference overhead. The results show that finite reranker feedback can be amortized across a query collection without retraining or rebuilding the retriever.