Visual document retrieval improves by picking tokens after query arrives
Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrieval
Information RetrievalArtificial Intelligence
Summary
Finding relevant images of documents quickly is hard because pages store lots of detailed information, which slows things down and uses lots of storage space. The authors study a two-step method: first, they use a small index to pick some promising pages, then they carefully choose which pieces of those pages to look at based on the actual search query. This method chooses tokens (small units of information) smartly, which keeps the search accurate but much faster. Their tests show this adaptive approach works better than just compressing data without considering the query.
What this means in practice
- •For digital library engineers: Improve speed and accuracy of searching scanned documents by dynamically selecting relevant tokens based on user queries.
- •For enterprise search developers: Enable efficient retrieval of image-rich documents by integrating query-aware token selection in existing search solutions to reduce compute and storage costs.
Authors
PS Rishi, Rajeev Ranjan Dwivedi, Vinod K Kurmi
Abstract
Late-interaction visual document retrievers preserve fine-grained page evidence by storing many token embeddings per page, but the resulting storage and query-time interaction costs make large-scale deployment expensive. Pooling document tokens before indexing offers a natural remedy, yet static pooling must decide which visual evidence to preserve before the query is known. We study an alternative: a heavily compressed hot-path index generates candidates, after which query-aware token budgeting operates on the original token sets of the shortlisted pages. We formulate this stage-two selection as a budgeted MaxSim coverage problem, show that a clipped version is monotone submodular, and compare coverage-only, cluster-guided, token-wise, and marginal-gain policies. On ten ViDoRe tasks with ColModernVBERT, direct static pooling reduces macro normalized discounted cumulative gain at rank five from 0.6309 without compression to 0.4738 at a thirty-two-fold pool factor. Under the same candidate-generation regime and a pool-factor-eight-equivalent reranking budget, token top-k recovers 93.93 percent of the full-token score, while greedy marginal-gain selection recovers 98.39 percent. Held-out and leave-one-dataset-out evaluations yield positive greedy improvements over token top-k on every dataset. The latency analysis reveals two useful operating points: token top-k for interactive retrieval and the naive greedy implementation as a quality upper envelope. Together, these results show that late-interaction visual retrieval benefits from query-aware allocation rather than query-agnostic pooling alone.