Alignment emerges between language and vision model representations
Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis
Machine Learning
Summary
When AI systems learn to understand language or images, they create internal ways to represent information. This paper looks at whether these internal representations become similar across different types of AI, like language and vision models. The authors found that as AI models get better, their internal understanding becomes more aligned, especially when the language descriptions are clear. However, the similarity between models of different types is still less than that seen between models of the same type. They also show that it’s important to compare fairly using proper baselines to avoid misleading results.
What this means in practice
- •For ai model developers: Improve multimodal AI models by understanding where and how internal representations align across language and vision networks, guiding better model integration.
- •For machine learning engineers: Use appropriate baselines to evaluate representation similarity in neural networks, avoiding misleading conclusions when comparing different model architectures.
Authors
Julia Herbst, Simon Roschmann, Quentin Bouniot, Zeynep Akata
Abstract
The Platonic Representation Hypothesis (PRH) suggests that neural networks are converging toward similar representation spaces. As language models become more capable, their representations are aligning with those of vision models. However, recent work has questioned whether this trend continues to hold for newer language models. Moreover, it has been shown that representation similarity metrics can be confounded by model depth and width, requiring appropriate baselines for interpretation. In this work, we first investigate where alignment emerges between unimodal models across model layers and representation dimensions. Next, we find that the PRH capability-alignment trend persists for newer models as long as the captions are sufficiently clean and informative. Finally, we contextualize cross-modal alignment, introducing unimodal and multimodal baselines on various datasets. Our results show that the cross-modal representational alignment of unimodal models can surpass that of explicitly aligned vision-language models, but remains substantially behind the alignment observed for models of the same modality. Code is available at https://github.com/ExplainableML/prh-closer-look.