LoRA methods improve low resource language translation but hurt reading skills
MoE--LoRA Methods for Multilingual Adaptation: A Systematic Empirical Study
Computation and LanguageMachine Learning
Summary
Large language models tend to perform much better in English than in many other languages. The authors studied how special add-ons called LoRA adapters affect these models when supporting over 200 languages. They found that LoRA adapters help translation work, especially for languages with fewer resources, but make reading comprehension worse. Also, the best LoRA setups are not easy to predict, so these adapters are better for specific goals rather than general improvement.
What this means in practice
- •For machine translation teams: Improve translation quality for low-resource languages by applying LoRA adapters to large language models.
- •For natural language processing engineers: Use LoRA adapters selectively to boost model performance in translation tasks without general multilingual capability losses.
Authors
Nikit Srivastava, Joel Dag, René Speck, Axel-Cyrille Ngonga Ngomo
Abstract
Routed low-rank adapters offer a parameter-efficient means to improve the performance of large language models (LLMs) across tasks. Hence, it is assumed that they can also be used to improve the support of low-resource languages in LLMs, which currently serve English far better than the thousands of other languages spoken worldwide. However, this assumption has never been verified empirically under a shared multilingual evaluation protocol. We address exactly this gap by comparing eight low-rank adaptation (LoRA) methods on a frozen base model using 200 unique languages in more than 9{,}000 evaluation settings. Our results suggest that a reduced-scale search can identify viable configurations of LoRA methods but does not reliably predict their relative performance, as the top-ranking configuration is never best at scale. LoRA reliably improves language-model fit on translation data, consistently degrades reading comprehension, and benefits low-resource languages most. These findings suggest that LoRA methods should be regarded as targeted tools rather than general-purpose means for improving the multilingual capabilities of LLMs.