Multilingual training adapts to changing weakest language dynamically
When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM
Computation and Language
Summary
Training multilingual AI models often struggles because some languages are harder to learn and get less attention during training. The authors introduce ARMOR-DP, a system that watches which language is currently hardest to learn and adjusts how much the model focuses on it during training. They show that this dynamic adjustment works better than always protecting a fixed language, helping the model learn languages more evenly. This approach treats the hardest language as something that can change over time, not a fixed problem.
What this means in practice
- •For natural language processing teams: Adapt training data sampling dynamically to improve balance across multiple languages in multilingual models.
- •For machine learning platform engineers: Integrate ARMOR-DP to monitor and adjust language focus during training, reducing wasted training on already-learned languages.
Authors
Hanyu Liao, Yijing Chen, Tao Ji
Abstract
Multilingual BabyLM systems usually fix language sampling proportions even though the weakest language can change during training. We present ARMOR-DP, a modular framework that diagnoses imbalance arising from representation, exposure, capacity, and objective pressure, and uses per-language validation loss to update sampling. Controlled experiments show that equal character budgets create a severe representation artifact for Chinese; byte-level encoding removes much of it, while static Chinese protection can subsequently make English the weakest language. The dynamic controller tracks these transitions and achieves a better score--balance tradeoff than fixed protection. On official evaluation, curriculum-dynamic checkpoints outperform a matched static Chinese-protective baseline while using fewer Chinese epochs; scaling capacity yields the strongest internal ARMOR-DP system. These results support a simple conclusion: multilingual weakness is a training-time state rather than a fixed property of a language. ARMOR-DP therefore separates diagnosis from intervention, allowing protection to follow the current bottleneck instead of a predetermined language label.