UEmbed: Unified Sparse and Dense Multimodal Embeddings
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageInformation Retrieval
AI summaryⓘ
The authors present UEmbed, a new model that combines both sparse (word-based) and dense (meaning-based) search methods in one efficient system. Unlike previous models that required complex setups for handling multiple types of data, UEmbed uses a simple decoder approach to handle text and images together. It splits the vocabulary into parts and predicts sparse scores for each, while also generating dense embeddings, all in a single pass. Tested on public datasets, UEmbed performs well compared to other models, showing promise for improving how search engines and AI assistants find information.
Sparse retrievalDense embeddingsMultimodal embeddingDecoder-only modelLexical matchingCausal hidden stateVocabulary partitioningMMEB-v2BEIR benchmarkRetrieval-augmented generation
Authors
Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Zhijie Nie, Yilun Zhao, Shu Wu
Abstract
Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.