MoEMB improves multimodal embeddings using efficient expert scaling
MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models
Machine LearningArtificial Intelligence
Summary
Handling many types of data like images and text together is tricky because embedding models often become too big or slow. The authors show a new way to make these models bigger and smarter without needing much extra computing power. They do this by splitting the work among different 'experts' inside the model, only using some experts for each task. This approach beats older methods while using less computing effort. They also study how to make the model use computing resources better by adjusting which experts are active as needed.
What this means in practice
- •For search engine developers: Improve large-scale multimedia search by using expert-based embeddings that reduce computation and speed up retrieval without losing accuracy.
- •For recommendation system engineers: Build efficient recommendation systems that handle diverse data types with scalable embeddings that adjust computational load dynamically.
Authors
Xuanming Cui, Shlok Kumar Mishra, Wentao Bao, Aashu Singh, Zihao Wang, Xiangjun Fan, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng
Abstract
Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size, retrieval effort, or scales the encoder into a heavy multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via reasoning tokens. However, embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose MOEMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoE-based UME, MoEMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only 3B active parameters, MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based embedding models towards large-scale retrieval and recommendation systems.