Disjoint tokens block knowledge sharing in multilingual language models
Why Pretraining Fails to Share Cross-Lingual Knowledge
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Large language models can understand many languages but often fail to share knowledge between them like humans do. The authors found that when languages use separate token representations, even identical copies of the same language keep their knowledge separate. This separation is a big reason why these models struggle to transfer what they learn across languages. They showed that making token representations shared—for example, by translating words directly—helps models share knowledge better and learn more efficiently.
What this means in practice
- •For multilingual ai developers: Improve multilingual model training by aligning token representations across languages to boost cross-lingual knowledge sharing and efficiency.
- •For machine translation engineers: Use shared token spaces created by word-wise translation to enhance the transfer of language understanding between language pairs in translation systems.
Authors
Adam Gaber, Uriel Dolev, Elisabeth Fittschen, Bobby Cheng, Yuval Marton, Leshem Choshen
Abstract
Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6\% of native-language learning efficiency --- 14$\times$ the baseline.