Language models represent translations similarly in middle layers
A theory of platonic representations in language models
Machine LearningComputation and Language
Summary
Language models trained on multiple languages show that the middle layers represent translated sentences in very similar ways, but this wasn’t fully understood before. The authors propose a theory that data have hidden structures shared across languages at abstract levels but different details on the surface. They use synthetic languages to prove that an ideal predictor’s internal messages explain this similarity pattern. Their theory also clarifies when models truly align languages in the same coordinates and predicts a way to increase cross-language similarity in real models.
What this means in practice
- •For multilingual model developers: Guide architecture design by explaining why shared representations peak in middle layers and how training data affects language similarity.
- •For machine translation engineers: Identify layer-specific features to improve cross-language alignment and model quality by using insights about shared abstract structures.
Tested on simulated data.
Authors
Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner, Daniel J. Korchinski, Santiago Acevedo, Matthieu Wyart
Abstract
Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.