Method audits missing documents in real-time retrieval systems
Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines
Computation and Language
Summary
Retrieval-augmented generation mixes searching documents and creating answers, but it’s hard to check if the search part finds all important documents. The authors introduce Re:CAP, a method that asks questions to find missing topics in the search results and uses a language model to judge if new documents add fresh information. Their method finds many relevant documents missed by common search approaches and works reliably on different datasets. This helps make sure that computer systems giving answers are not missing key facts from their sources.
What this means in practice
- •For search engine engineers: Improve monitoring of real-time document search quality by identifying missing relevant documents without exhaustive relevance labels.
- •For enterprise knowledge management teams: Audit document coverage in corporate retrieval systems to ensure answers include diverse and previously unretrieved information.
Authors
Aviral Joshi, Hanoz Bhathena, Max Nelson, Saket Sharma
Abstract
Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline's initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9-29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss $κ$ = 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within $\pm$1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.