Multilingual AI struggles more with low-resource languages after fine-tuning
Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models
Computation and Language
Summary
Measuring how well language models understand grammar depends a lot on how you test them. The authors found that when large language models are further trained, their grammar skills get worse, especially for languages with fewer resources. However, these models still know more grammar than they show unless questioned carefully in the right language. Using prompts in a language's native tongue helps reveal hidden knowledge, particularly for less common languages. The study suggests using different evaluation methods tailored to each language to better assess language models fairly.
What this means in practice
- •For natural language processing teams: Develop evaluation methods that reveal true grammar knowledge in low-resource languages by combining multiple testing strategies and native-language prompting.
- •For machine translation developers: Improve fine-tuning processes by monitoring how post-training affects grammar skills across languages to avoid degrading performance in less-resourced languages.
Authors
Zhanyu Chen, Jaap Jumelet
Abstract
Claims about the grammatical competence of multilingual language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm, post-training, and language resource availability has not been systematically examined. We evaluate base and post-trained models from six families on MultiBLiMP, a syntactic minimal-pair benchmark covering 101 languages, using four evaluation methods. We report three principal findings. First, post-training degrades grammatical competence, but the magnitude of this effect is reduced unevenly by model scale, while low-resource languages bear the highest cost. Second, post-trained models retain grammatical knowledge they cannot articulate through explicit prompting, yet this is measurable only in high-resource languages, because near-chance baselines in low-resource settings leave little knowledge to hide. Third, native-language prompting recovers otherwise hidden competence on low-resource languages, demonstrating that only high-resource languages can be probed directly from unprompted probabilities. We conclude that multilingual grammatical evaluation must adopt language-informed, multi-paradigm protocols to avoid systematically underestimating low-resource abilities.