Adaptive sampling improves multilingual model training across languages
Online Language Adaptive Sampling for Better Distributed Cross-lingual Gains
Computation and LanguageArtificial Intelligence
Summary
Multilingual language models sometimes struggle to learn equally well across many languages, especially those with less training data. The authors found that instead of randomly choosing sentences from all languages, adjusting how often each language is shown during training can help. Their method automatically increases training focus on languages that are harder to align, which helps the model improve overall and better across all languages. They tested this with popular language models and saw consistent performance gains.
What this means in practice
- •For natural language processing engineers: Improve multilingual model training efficiency by dynamically focusing on languages with poorer alignment during cross-lingual transfer tasks.
- •For machine translation developers: Enhance translation quality for low-resource languages by adaptively increasing representation of these languages during model training.
Authors
Quang Phuoc Nguyen, Félix Gaschi, David Anugraha, Santiago Martínez Novoa, En-Shiun Annie Lee
Abstract
Realignment is a promising approach for improving the cross-lingual transfer ability of multilingual language models, particularly for extremely low-resource languages (LRLs). However, existing realignment methods rely on uniform and random sampling of parallel sentences across languages, which may be suboptimal under limited batch sizes. In practice, models may benefit from seeing certain languages more frequently, especially those that are poorly aligned, and the optimal distribution can evolve throughout training. In this work, we propose a simple yet effective adaptive sampling strategy that assigns trainable sampling probabilities to each language. Languages that contribute more to the realignment loss are sampled more frequently in subsequent batches, and the optimal distribution can evolve throughout training. Our method employs an inner-outer optimization loop with a small overhead, leading to consistent performance improvements and, more importantly, distributing the gains across languages. We observed a $+0.67$ average performance increase on all tasks with XLM-R, and $+0.60$ with Gemma 2 9B compared with uniform realignment. Furthermore, our method is robust across different models. Code available at https://github.com/felixgaschi/multilingual-alignment-and-transfer.