Models trained on different data can align without paired examples
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Models that understand different kinds of data, like images and text, usually need examples that match across these types to work together well. The authors found that models trained separately can still align their understanding using shared geometric patterns without needing any paired examples. They developed a straightforward method that matches these models by looking at their overall shape instead of pairs. This method helps improve tasks like generating images from text, even when matching examples are scarce or missing.
What this means in practice
- •For multimedia software developers: Enable text-to-image features without needing large aligned text-image datasets by aligning pretrained models from different modalities.$Commercial implications: Enables development of products generating images from text without costly paired data, useful for apps and creative tools.
- •For machine learning engineers: Reduce reliance on costly paired data by using geometric alignment to connect independently trained models for various cross-modal tasks.
Authors
Dominik Schnaus, Thomas Dagès, Daniel Cremers, Xi Wang, Phillip Isola
Abstract
Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.