Mixture-of-experts language models improve search speed and accuracy
Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers
Information RetrievalComputation and Language
Summary
Searching through large amounts of text quickly and accurately is hard because big language models take a lot of computing power. The authors studied a special kind of model called mixture-of-experts (MoE) that only uses part of its brain when answering, making it faster. They found these MoE models can find relevant information better and more efficiently than regular models of similar size. Also, you can reduce how much of the model is active during search without losing much accuracy. This means MoE models can make searching smarter and quicker at the same time.
What this means in practice
- •For search infrastructure engineers: Create faster and more accurate first-step search systems by integrating MoE language models that use fewer parameters per query.
- •For cloud service operators: Reduce the computational cost and latency of language model-based retrieval services by deploying MoE models that selectively activate parts for each query.
- •For enterprise search developers: Build efficient document retrieval features in large-scale enterprise applications using MoE retrievers that match or exceed dense model accuracy.$Commercial implications: Allows development of commercial search products delivering better accuracy with lower computational resources, reducing operational costs.
Authors
Anubhav Shrestha, Safal Shrestha, Minwu Kim, Torsten Suel, Keith Ross
Abstract
Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, evaluating them across diverse datasets, and measuring query encoding time under the same serving configuration. We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR. One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time. We further show that the number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%. Recent rerankers provide only modest additional gains over strong MoE first stages, which often match or exceed the reranked configurations we evaluate. Together, these results show that MoE LLMs can be strong and efficient first-stage retrievers.