Efficient document search improves speed and accuracy with cascaded retrieval
Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search
Computation and Language
Summary
Searching for information in documents with lots of pictures, tables, and complex layouts is hard and slow when the collection is large. The authors created SCOUT, a two-step method that first quickly narrows down the search to promising pages and then carefully rechecks just those. SCOUT also uses summaries of the documents to improve understanding and combines both visual and semantic clues to get better results. This approach finds relevant pages much faster and more accurately than previous methods.
What this means in practice
- •For enterprise search teams: Improve large-scale document retrieval systems to deliver relevant results faster by using cascaded visual and semantic search with query refinement.
- •For legal document analysts: Accelerate searching through complex legal documents containing tables and figures by combining visual and semantic cues to find key information efficiently.
Authors
Diana Turmakhan, Aseem Srivastava, Vivek Gupta, Iryna Gurevych, Thamar Solorio
Abstract
Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layout-dependent structures. Current vision-language retrievers address this but incur query-time costs that grow with corpus size and are impractical to scale. We propose SCOUT, a cascaded two-track retrieval framework that separates retrieval performance from exhaustive search. The visual track first uses a lightweight parent retriever to filter the corpus to a compact candidate pool, then applies a stronger multi-vector reranker only to those candidates. In parallel, a semantic track retrieves over hierarchical page and document summaries generated offline to provide a signal complementary to visual matching. The outputs are fused through cross-modal query refinement, which uses semantic scores to adjust the visual query representation before final ranking. We evaluate SCOUT on 4 major benchmarks against 6 baselines and observe a significant query-time latency improvement up to 80.7x (28.8x on average) relative to the best baseline, while exceeding state-of-the-art retrieval performance by up to 7.8% NDCG across all benchmarks.