Cross passage pretraining improves document retrieval accuracy
xPRT: Cross-Passage Retrieval Pretraining
Information Retrieval
Summary
Finding the best documents to answer questions or queries often needs lots of labeled examples, which are hard to get. The authors introduce a way to teach retrieval models by learning from connections and important parts spread across whole documents, without needing labeled data. Their approach, called xPRT, improves retrieval performance consistently compared to other methods. This shows that single documents hold more useful information for training than previously thought.
What this means in practice
- •For search engine developers: Improve search results by training retrievers on unlabelled documents, capturing richer document context for better query matching.
- •For chatbot builders: Enhance retrieval-augmented generation systems by pretraining retrievers to use connections across document parts, improving answer accuracy.
Authors
Danush Khanna, Himanshu Singhal, Pranjal A. Chitale, Bishal Santra
Abstract
Dense retrievers are a foundational component underlying modern RAG and search systems. Training these models typically requires a curated collection of annotated query-document pairs, which are prohibitively expensive to obtain outside a handful of well-resourced domains. This has motivated a line of work that leverages large-scale unsupervised retrieval pretraining from unlabelled text alone. Most such methods learn by optimizing a language modelling or contrastive learning objective over short spans within a document. This raises the question: how can we extract richer learning signals from a single document for retrieval pretraining? To address it, we introduce xPRT (Cross-Passage Retrieval Pretraining), an unsupervised framework that learns from the key information distributed across an entire document and the connections between its parts. We evaluate xPRT against five state-of-the-art objectives on MTEB English v2, continuing pretraining from a public unsupervised checkpoint while holding the corpus, schedule, and training procedure fixed. Across diverse tasks spanning all task types, we observe that xPRT consistently outperforms the strongest baseline by 3.2% on an average. Notably, in this continual pretraining setting, many baseline methods incur substantial performance regressions, whereas our method delivers the largest complementary gains over the initial pretrained checkpoint. Controlled ablations confirm that xPRT's performance gains stem from its proposed cross-passage signals. xPRT demonstrates that a single unlabelled document is a much richer source of retrieval signal than prior work assumed and offers complementary gains to what current retrievers already capture.