Open source improves reproducibility and testing of document search methods
REDSI: Addressing the Reproducibility and Evaluation Consistency of Differentiable Search Indexing for Document Retrieval
Information Retrieval
Summary
Finding reliable ways to search documents using AI is tricky because previous methods were hard to copy exactly and gave varying results. The authors created ReDSI, an open-source version that supports all main ways of identifying documents and clearly explains how to prepare the data used. Their tests show ReDSI works as well as or better than earlier versions. They also explored how making models smaller or changing training affects performance, suggesting new ideas for future research in document search.
differentiable search indexdocument retrievalreproducibilityopen sourceNatural Questions datasetparameter efficiencytraining methodsdecoding strategiesgenerative retrievalmodel downscaling
Authors
Vivien Nicolas, Hicham Randrianarivo, Pascale Sébillot, Caio Corro
Abstract
The differentiable search index (DSI) framework (Tay et al., 2022) has become the de facto baseline for generative retrieval. However, DSI is hard to reproduce: no public implementation covers all three original document identifier types (atomic, naive, semantic), reported results vary widely, and the ubiquitous NQ320K dataset is built from Natural Questions through diverse and underspecified preprocessing. We introduce ReDSI, the first open-source DSI implementation supporting all three identifier types, together with a parameterizable and well-documented NQ320K construction pipeline. Experimentally, we achieve results that are competitive with or stronger than previous DSI baselines. Moreover, we conduct extensive experiments under model downscaling, covering retrieval effectiveness, parameter efficiency, training methods and decoding strategies, opening novel directions for future research.