Document retrievers compared across domains reveal trade offs and performance gaps
A Systematic Multi-Domain Evaluation of Document Retrievers
Information Retrieval
Summary
Document retrievers help computers find relevant information in large collections, which is important for many AI applications. The authors compare 33 different document retrieval methods from three categories on seven datasets without extra tuning, to see how well they work and what problems they have. They found that some methods perform best overall but are slower, while others are faster but slightly less accurate. The study highlights differences in where these methods tend to fail, suggesting ways to improve future document retrieval systems.
What this means in practice
- •For search engine engineers: Choose retrieval models based on speed and accuracy trade-offs for diverse content and queries without expensive tuning.
- •For ai product developers: Incorporate off-the-shelf document retrievers into AI systems to improve information access across multiple domains and tasks.
Authors
Valentin Velev, Andreas Spitz
Abstract
Document retrieval is a crucial component of many modern AI systems, directly influencing their effectiveness, robustness, and fairness in downstream tasks. While recent years have seen a growing number of retrievers, comparative studies in the literature are typically limited in scope or focused on singular benchmarks, domains, or model families. This fragmentation makes it difficult to draw reliable conclusions about the relative strengths, weaknesses, and trade-offs of document retrievers. To address this gap, we conduct a large-scale empirical evaluation of document retrievers, covering three families (sparse, dense, and expansion-based) and evaluating 33 retrievers across seven IR datasets, analyzing retrieval quality, runtime, and failure points. Rather than tuning each model individually, we evaluate every retriever off the shelf, under the configuration reconstructable from its public documentation and a uniform compute budget. Our results show that NV-Embed-v2 achieves the strongest performance on four of the seven datasets, albeit at the cost of substantial query latencies. Among sparse retrievers, we find that SPLADE-v3 rivals the top-performing approach despite much lower latency, and even achieves top scores on MS MARCO. On instruction-following datasets, GritLM delivers the best performance. Finally, an analysis of the retrievers' failure points reveals contrasts between models and families that indicate potential for unrealized gains in retrieval performance.