Open source tool improves consistency and results in document retrieval
REDSI: Addressing the Reproducibility and Evaluation Consistency of Differentiable Search Indexing for Document Retrieval
Information Retrieval
Summary
Finding information in large document sets often uses special tools that learn by example, but these tools can be hard to recreate and compare. The authors introduced ReDSI, a new open-source tool that fixes these issues by supporting all original methods and clearly showing how to prepare a popular dataset. Their tests showed that ReDSI works as well or better than prior tools, and they explored how smaller models and different training methods affect results. This work helps make research on document searching more reliable and easier to build upon.
What this means in practice
- •For search engine developers: Use the open-source ReDSI tool to build more reproducible and consistent document retrieval systems.
- •For machine learning engineers: Experiment with smaller models and training methods supported by ReDSI to optimize retrieval performance and resource use.
Authors
Vivien Nicolas, Hicham Randrianarivo, Pascale Sébillot, Caio Corro
Abstract
The differentiable search index (DSI) framework (Tay et al., 2022) has become the de facto baseline for generative retrieval. However, DSI is hard to reproduce: no public implementation covers all three original document identifier types (atomic, naive, semantic), reported results vary widely, and the ubiquitous NQ320K dataset is built from Natural Questions through diverse and underspecified preprocessing. We introduce ReDSI, the first open-source DSI implementation supporting all three identifier types, together with a parameterizable and well-documented NQ320K construction pipeline. Experimentally, we achieve results that are competitive with or stronger than previous DSI baselines. Moreover, we conduct extensive experiments under model downscaling, covering retrieval effectiveness, parameter efficiency, training methods and decoding strategies, opening novel directions for future research.