Adaptive document retrieval improves answer accuracy and efficiency
Pre-retrieval Query Clustering for Adaptive Top-k Document Retrieval in RAG Systems
Information Retrieval
Summary
Many AI systems that answer questions fetch the same number of documents, which can either bring in too much useless info or miss important facts. The authors introduce a way to guess in advance how many documents to fetch based on what kind of question is asked. They group similar questions and assign a recommended number of documents to retrieve for each group, making the process faster and more accurate. Tests show this method improves answer quality and reduces unnecessary data without losing accuracy.
What this means in practice
- •For enterprise search engineers: Select an optimal number of documents to retrieve per query, improving accuracy and reducing retrieval costs in large corporate databases.
- •For legal tech developers: Optimize document retrieval depth for complex legal queries to prevent missing important information while controlling query processing costs.
Authors
Ye Xia, Emre Yamangil, Haixun Wang
Abstract
RAG systems commonly retrieve a fixed number of documents (top-k) to ground generation, but this static approach is brittle: simple queries suffer over-retrieval (adding noise and cost) while complex queries are under-retrieved, causing recall failures that cascade into incorrect answers. Motivated by the question of how many documents must be retrieved to answer an arbitrary query reliably, we propose a practical, general framework for query-adaptive retrieval depth. Offline, we estimate per-query retrieval difficulty by measuring NDCG under the default retriever and deriving a query-specific "saturation" point k* from the NDCG-k curve. Because computing these signals online is expensive, we cluster a large set of queries in embedding space and summarize each cluster with a recommended retrieval depth that targets high coverage (e.g., \textasciitilde{}95\%) using a mean-plus-variance rule. At runtime, the system assigns an incoming query to a cluster and selects the corresponding top-k in constant time. Compared with post-retrieval confidence methods that rely on clustering retrieved documents, our approach is pre-retrieval and query-centric, making it robust in heterogeneous, case-like corpora and applicable across domains such as legal, healthcare, finance, and enterprise search. Finally, this framework has been tested in full-traffic queries that improved $F_1$ by over 36\% while reducing token usage by 14\% on low-complexity clusters without accuracy loss.