Retrieval diversity improves answers only when evidence is repeated
Finding the Right Balance: Relevance and Diversity in LLM Retrieval
Information RetrievalComputation and Language
Summary
Finding the right balance between relevant and diverse search results is tricky. This paper shows that adding diverse results can actually hurt answer quality when search results are unique but helps when important information is repeated in multiple places. The authors create a rule that decides when to use diverse searches based on how many unique documents are found and how many different pieces of evidence the query needs. Their approach improves how answers are found, especially for questions that require multiple sources of evidence.
What this means in practice
- •For chatbot developers: Improve chatbot answer quality by selectively diversifying retrieved documents based on query evidence needs and document uniqueness.
- •For search engine engineers: Enhance search relevance by automatically detecting when to diversify retrieved results to improve coverage on queries requiring multiple evidence sources.
Authors
Guillaume Brouillette, Faustin Kagabo, Usef Faghihi, Nadia Ghazzali
Abstract
Retrieval diversification is widely available in retrieval-augmented generation (RAG) frameworks, yet prior studies disagree on whether it improves retrieval and answer quality. We show that its effectiveness varies primarily with candidate-pool redundancy, in a pattern consistent with the number of distinct evidence pieces a query requires. Using controlled near-duplicate injection and production-style overlapping chunking, we find that diversification harms relevance, evidence coverage and answer quality on clean pools, but becomes beneficial on multi-evidence tasks when redundancy causes nearest-neighbor retrieval to select repeated passages. We therefore introduce a query-adaptive rule that diversifies only when the effective number of distinct documents in the nearest-neighbor top-$k$ selection falls below the query's evidence requirement. Computed from existing embeddings, the rule captures most of the achievable gain, transfers across datasets and encoders and automatically reduces to nearest-neighbor retrieval for single-evidence queries. We also introduce RNG-Score, a geometric reranker with an exact nearest-neighbor fallback whose margin indicates duplicate structure. Overall, we conclude that diversification should be used selectively, based on observable redundancy and evidence requirements.