Relational structure helps language models connect meanings across languages
Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization
Computation and Language
Summary
Languages can be very different, so it’s hard for language models to understand things in one language by learning from another without shared words. The authors show that when two languages describe the same relationships in the world, like math operations, the model can align meanings across languages better. This happens because the relationships reduce the number of ways to match things, making some connections clearer. Their experiments with a simple bilingual Transformer show this alignment depends on the type of relationships, which also predicts how well the model transfers knowledge.
What this means in practice
- •For machine translation developers: Design translation systems that better transfer understanding without shared vocabulary by using relational constraints to align meanings.
- •For conversational ai teams: Improve multilingual dialogue systems by structuring knowledge around shared relations to enable cross-lingual responses without explicit token alignment.
Authors
Frederick Riemenschneider, Anette Frank
Abstract
Why do multilingual language models generalize across languages even without shared vocabulary? The Platonic Representation Hypothesis posits that models trained on different data, objectives, and modalities converge toward a shared statistical model of reality. Extending this intuition to representations inside a single multilingual model, we propose the Relational Identifiability Hypothesis: relations in a shared world can promote entity alignment by ruling out alternative cross-lingual correspondences, making corresponding entities more likely to become directly readable without an additional transformation. We test this hypothesis in a controlled bilingual Transformer whose two token-disjoint languages describe the same modular-arithmetic world. We find that the languages can develop a shared answer subspace without aligning corresponding numbers. Cross-lingual alignment emerges selectively when the available relations constrain number correspondences: subtraction makes zero task-level identifiable, inducing selective alignment, while an order-sensitive operation produces broad alignment across numbers. These patterns predict success in zero-shot cross-lingual transfer. Together, our results show that relational structure in a shared world can promote entity alignment and enable transfer.