Illuminating Visual Identity in Universal Multimodal Embeddings
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors focus on improving how machines can recognize and distinguish specific visual identities, like unique people or objects, within universal multimodal embeddings that combine different types of data. They identify that previous models did not handle this visual identity discrimination well, so they created a new large-scale benchmark called MVEB to help train and test this ability. Their method uses a special learning technique that balances general understanding and precise identity recognition. Experiments show their approach improves visual identity understanding without losing overall multimodal performance.
Universal Multimodal EmbeddingsMultimodal Large Language ModelsVisual Identity DiscriminationInstance RetrievalRe-identificationIdentity-aware SamplingBenchmark DatasetMultimodal RepresentationSynthetic DataEmbedding Space
Authors
Jiawei Cao, Junyi Feng, Jiashen Hua, Ziheng Huang, Bing Deng, Kaijie Wu, Chaochen Gu, Jieping Ye
Abstract
Universal Multimodal Embeddings (UMEs) aim to unify various modalities and tasks into a shared representation space. In recent years, this field has witnessed substantial progress driven by the development of Multimodal Large Language Models (MLLMs). However, a crucial capability, visual identity discrimination, remains underexplored in existing UME methods, despite its critical role in a wide range of tasks, including instance retrieval, re-identification, and identity preservation in AI-generated content. To bridge this gap, we propose a unified formulation for visual identity discrimination~(VisID) and introduce $\textbf{MVEB}$ ($\textbf{M}$ultimodal $\textbf{V}$isual Identity $\textbf{E}$mbedding $\textbf{B}$enchmark), a large-scale benchmark curated from both real-world and synthetic datasets to support evaluation and training. Furthermore, we present a simple yet effective learning framework that jointly optimizes general multimodal and visual identity representations through a carefully designed identity-aware sampling mechanism. Extensive experiments demonstrate that our approach successfully endows UMEs with strong identity discrimination capability and maintains competitive general multimodal performance. We believe this work not only illuminates a critical yet neglected capability, but also takes a step toward more holistic universal multimodal embeddings. Code and data are available at \href{https://chrisclear3.github.io/MVEB}{MVEB}.