Unimodal training losses guide cross-modal alignment in AI models
Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment
Machine Learning
Summary
Sometimes computers learn to connect different types of information, like pictures and text, even when trained separately. This paper shows that when computers train on just one type of data using certain methods, it naturally helps them learn to line up information across types. The authors explain why this happens by linking a common training technique to a way of measuring how well these connections form. They tested their ideas on many different models and found consistent evidence that training on single data types can align different senses like vision, language, and sound.
What this means in practice
- •For multimodal ai engineers: Design unimodal training methods that improve cross-modal embeddings without joint training data.
- •For audio processing developers: Use unimodal augmentation objectives to align audio features with text or visual features in frozen models more effectively.
Authors
Zohar Mashiah, Nir Ben-Ari, Ronen Talmon, Uri Shaham
Abstract
Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.