Progressive embedding training improves multimodal similarity models
PACE: Progressive Angular-to-Norm Contrastive Embedding
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageMachine Learning
Summary
Comparing different types of data like images and text involves turning them into shared codes that can be compared. Many current methods focus only on the angle between these codes, ignoring their length, which can carry extra meaning. The researchers found that directly using the full code similarity can be unstable and less effective at first. They proposed a two-step training method to first build a stable angle-based code and then include length for richer meaning, leading to better performance. They also designed a new way to focus training on harder examples, improving the overall learning process.
What this means in practice
- •For machine learning engineers: Train multimodal similarity models that better use embedding angles and lengths for improved retrieval accuracy.
- •For software developers: Implement a two-stage training routine to stabilize and enhance embedding models in apps combining images, text, and other data.
Authors
Yanping Li, Wei Zhou, Yawen Liu, Yibo Wang, Ke Zhu, Guangda Huzhang, Qing-Guo Chen, Zhao Xu, Jun Zhang, Wei Wei
Abstract
Multimodal embedding models encode heterogeneous inputs into a shared embedding space, enabling efficient similarity computation across modalities and tasks. Most existing methods optimize cosine-based contrastive objectives, which promote stable training but restrict semantic compatibility to angular geometry, precluding embedding norms from serving as an additional semantic signal. However, directly optimizing the more expressive dot-product similarity, which leverages both angular and norm information, underperforms cosine-based training and exhibits unstable training dynamics. We attribute this discrepancy to premature optimization-space expansion, manifested as angular--norm entanglement and directional anisotropy in the representation space and further compounded by full-parameter fine-tuning. In this paper, we propose PACE, a two-stage framework that progressively expands both the representation and trainable parameter spaces. Stage I combines cosine-based objective with low-rank adaptation to establish a reliable angular geometry within constrained optimization spaces. Stage II switches to dot-product similarity and full-parameter fine-tuning, enabling embedding directions and norms to jointly encode semantic information. We further introduce Focal Embedding Loss, a confidence-adaptive objective that downweights queries with high positive retrieval confidence while emphasizing ambiguous queries with competitive negatives. Experiments across multiple backbone scales and diverse multimodal embedding tasks consistently validate the effectiveness of PACE.