Language models partially transfer sentence structure between languages
Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models
Computation and Language
Summary
Language models sometimes learn grammar rules from one language and use them when learning another, but it's not perfect. The researchers tested how well models transfer knowledge about sentence structure, especially hierarchy, from one language to another. They found models do share some understanding, but it depends a lot on the languages and how much their vocabularies overlap. This means language models only somewhat learn universal grammar principles when trained across languages.
What this means in practice
- •For multilingual ai developers: Improve model training by focusing on vocabulary overlap to enhance grammar learning transfer between languages.
- •For machine translation engineers: Design training strategies that consider structural biases for better syntactic accuracy in translating less-resourced languages.
Authors
Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu Kano
Abstract
Language models (LMs) often exhibit successful cross-lingual transfer through continued (pre)training, still their underlying process remains unclear. One possibility is that LMs can leverage the evidence toward language-universal principles, namely, hierarchical structure, from a source language (L1) to a new language (L2), while many other non-linguistic explanations, e.g., improved initialization of position embeddings, can be potentially possible. In this study, to quantify the cross-linguistic transferability of linguistic evidence in cross-lingual LM training, we introduce a new experimental design, namely, a cross-lingual poverty-of-stimulus (CPOS) setting that ablates how LMs' hierarchical generalization in L2 can be led by structural evidence provided in L1. We contrast counterfactual L1 and L2 learning scenarios where the L2 corpus is ambiguous between linear and hierarchical grammar rules, while the amount of positive evidence for the hierarchical rule is controlled on the L1 side. Our experimental results show that LMs can partially, yet not perfectly, transfer L1 structural evidence to L2, yielding only a small but detectable hierarchical bias, while the strength of transfer depends on language pairs. We further ablate the conditions for successful transfer, motivated by recent reports on the substantial impact of vocabulary overlap. We indeed find that vocabulary overlap is crucial even for the cross-lingual transfer of hierarchical syntactic generalization. This suggests that this also explains the language-pair-dependent tendencies in the main experiments and suggests that the transfer of linguistic evidence in continued new-language learning is only conditionally achievable.