Multilingual LLMs keep language identity across all layers

Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere

Computation and Language

Summary

This paper looks at where language-specific features show up inside multilingual large language models (LLMs). Earlier studies suggested that languages are mostly identified in the very first and last layers of the model, with the middle layers being language-neutral. The authors tested this idea on a model called Llama 3.2 1B using several experiments with Portuguese and Spanish and one covering over 200 languages. They found that language identity is actually present throughout all the layers and not just at the beginning and end. They also noticed the model has a bias toward classifying ambiguous language samples as Spanish rather than Portuguese, which is different from how humans would decide.

What this means in practice

  • For ai system builders: Improve language detection components by using embeddings from all model layers, not just the first or last ones.
  • For multilingual service teams: Adjust language classification algorithms to account for biases between similar languages revealed by the model’s behavior.

Authors

Mikaela Martins, Jonathan Dunn

Abstract

Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
View PDFOpen arXiv