Cross-lingual alignment improves multilingual decoder-only language models
Cross-Lingual Alignment for Decoder-Only Models using MoE Routers
Computation and LanguageArtificial Intelligence
Summary
Aligning how language models understand different languages is hard when the model only generates text but doesn’t encode it separately. The authors found a way to improve this alignment by focusing on a special part of the model called the mixture-of-experts (MoE) routers. These routers help group parts of the input, making it easier to compare languages at a broader level. Their method improved the model’s ability to handle multiple languages better than before.
What this means in practice
- •For multilingual ai developers: Enhance multilingual language models for better translation and understanding by aligning MoE router outputs across languages during training.
- •For machine learning engineers: Implement cross-lingual alignment techniques that use MoE router outputs to improve transfer learning in decoder-only transformer architectures.
Authors
Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed, Aditi Khandelwal, Nanyun Peng
Abstract
Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only LLMs because of varying multilingual tokenization. However, a growing amount of research suggests that even in LLMs, higher cross-lingual representational alignment leads to improved cross-lingual transfer. In this paper, we propose a novel approach to reimagine cross-lingual contrastive learning given the architectural constraints of modern LLMs. Rather than applying an auxiliary alignment loss on hidden states, we propose using the outputs of the mixture-of-experts (MoE) routers as the target for alignment. Router outputs lend themselves better to pooling over many tokens, enabling more reliable cross-lingual comparisons at the sequence-level. Controlled continual pre-training experiments on four open-source MoEs show that incorporating this routing loss also aligns the underlying hidden representations across languages. Most importantly, this loss improves multilingual performance on our diverse evaluation suite, demonstrating the potential of cross-lingual MoE router alignment.