Compact multilingual retrievers match larger models for European languages
Parameter-Efficient Retrievers for Polish and European Languages
Computation and Language
Summary
Large language models help computers find relevant documents but are very big and slow, making them hard to use frequently or for many languages. This paper shows a new way to train smaller, faster search tools that work almost as well as the biggest ones, especially for Polish and other European languages. The authors used clever training techniques that don’t need lots of labeled examples but learn from existing strong models. They made several versions, some very small and some larger, and tested them widely, finding these compact models often do better than much bigger ones.
What this means in practice
- •For search engine developers: Create efficient multilingual search systems that handle frequent updates and long documents with lower computational costs.
- •For digital library teams: Implement compact retrievers to index and search large, diverse European language collections quickly and cost-effectively.
Authors
Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec, Michał Perełkiewicz
Abstract
Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a three-stage training pipeline for developing compact and efficient retrievers that remain competitive with substantially larger models. The pipeline combines cross-lingual alignment, relational knowledge distillation, and contrastive fine-tuning. It requires no original ground-truth relevance labels, relying exclusively on supervision generated by strong embedding models and rerankers utilised as teachers. Using this pipeline, we develop PolDense and EuroDense, both supporting contexts of up to 8,192 tokens. PolDense is a family of six Polish retrievers ranging from 17M to 1B parameters. EuroDense is a 435M-parameter retriever supporting nine European languages. We conduct an extensive evaluation covering 41 Polish and 150 multilingual retrieval tasks. The results demonstrate strong quality-efficiency trade-offs. PolDense-1B outperforms the evaluated retrievers with up to 9B parameters, while the PolDense family forms the Pareto frontier across model sizes. Among the evaluated models below 1B parameters, EuroDense ranks first in both task-averaged and language-averaged performance and leads in seven of nine languages. We release all models publicly.