DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search
2026-07-29 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors created a detailed, open method for training search models using a large set of English example pairs. They trained two types of models, one that looks at entire text pairs and another that compares parts of the text separately, both performing very well on English tasks. They then translated the training data into eight other languages and trained multilingual versions, finding that the model comparing parts of texts works better on languages it wasn’t trained on. This work helps others build and understand search models across different languages and they shared all their data and code publicly.
retrieval modelscontrastive learningdense retrievallate interactionBEIR benchmarktranslate-trainmultilingual modelshard negativesmmBERTnDCG@10
Authors
Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Amélie Chatelain
Abstract
State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts. This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe. We publicly release the models, datasets, and training code.