RidgeRank speeds up visual document reranking with score fusion
RidgeRank: Efficient Visual Document Reranking via Score Fusion and a Shallow Linear Readout
Information Retrieval
Summary
Sorting visual documents to find the most relevant ones can be slow when using powerful language models. The authors propose RidgeRank, a way to quickly combine scores from a fast retriever and a slower, more accurate reranker. By cleverly mixing these scores, RidgeRank keeps most of the accuracy but runs much faster. Tests on multiple datasets show it nearly matches full reranking accuracy with several times speedup.
What this means in practice
- •For search engine developers: Improve speed and accuracy of visual document search reranking by combining fast retriever and slower reranker scores with an optimal fusion approach.
- •For document management teams: Enable more efficient ranking of scanned or image-heavy documents with minimal loss in relevance quality during retrieval tasks.
Authors
Shubing Yang, Dongfang Zhao
Abstract
Multimodal language models rerank visual document retrieval results accurately, but scoring every candidate page at full cost makes them slow. Some methods that compress these rerankers need relevance labels to regain accuracy, and they rank by the reranker score alone. RidgeRank measures how much relevance signal the reranker score lacks and recovers it from the retriever score through a closed-form fusion rule. Maximizing a correlation objective gives the optimal fusion weight, along with the exact condition under which the reranker score by itself cannot reach that optimum. The reranker is further corrected by a single vector applied to an intermediate hidden state, obtained through one centered ridge regression onto the same model's full-depth scores on uncompressed pages. On 12 datasets drawn from ViDoRe 2 and ViDoRe 3, evaluated with two retrievers and two language model backbones, RidgeRank brings NDCG@5 to within 1.2 pp of a full cross encoder with speedups of up to 48 times, advancing the accuracy and latency Pareto frontier for visual document reranking.