LoRA methods improve low resource language translation but hurt reading skills

MoE--LoRA Methods for Multilingual Adaptation: A Systematic Empirical Study

Computation and LanguageMachine Learning

Summary

Large language models tend to perform much better in English than in many other languages. The authors studied how special add-ons called LoRA adapters affect these models when supporting over 200 languages. They found that LoRA adapters help translation work, especially for languages with fewer resources, but make reading comprehension worse. Also, the best LoRA setups are not easy to predict, so these adapters are better for specific goals rather than general improvement.

What this means in practice

Authors

Nikit Srivastava, Joel Dag, René Speck, Axel-Cyrille Ngonga Ngomo

Abstract

Routed low-rank adapters offer a parameter-efficient means to improve the performance of large language models (LLMs) across tasks. Hence, it is assumed that they can also be used to improve the support of low-resource languages in LLMs, which currently serve English far better than the thousands of other languages spoken worldwide. However, this assumption has never been verified empirically under a shared multilingual evaluation protocol. We address exactly this gap by comparing eight low-rank adaptation (LoRA) methods on a frozen base model using 200 unique languages in more than 9{,}000 evaluation settings. Our results suggest that a reduced-scale search can identify viable configurations of LoRA methods but does not reliably predict their relative performance, as the top-ranking configuration is never best at scale. LoRA reliably improves language-model fit on translation data, consistently degrades reading comprehension, and benefits low-resource languages most. These findings suggest that LoRA methods should be regarded as targeted tools rather than general-purpose means for improving the multilingual capabilities of LLMs.