New method identifies language neurons more precisely in multilingual AI models
Distribution-aware Language Neuron Identification in Multilingual Large Language Models
Computation and Language
Summary
Multilingual AI models use certain neurons that respond mostly to specific languages. Previous methods identified these neurons by checking if their activation was simply positive or not. The authors propose a better way by looking at the full range of neuron activations across languages and how these activation patterns overlap. This new method can more accurately find neurons that are truly language-specific, which helps understand how multilingual models process different languages.
What this means in practice
- •For multilingual ai developers: Improve neural network interpretability by better isolating language-specific neurons within multilingual language models.
- •For machine translation engineers: Enhance debugging and fine-tuning of language-specific components in translation models by applying more precise neuron identification methods.
Authors
Minjun Kim, Inho Won, Junghun Yuk, Dongyeon Kim, Jihyo Kim, KyungTae Lim
Abstract
Multilingual large language models (mLLMs) contain a small fraction of feed-forward neurons that are sensitive to particular languages, commonly termed language-specific neurons. Existing work measures language specificity using the entropy of each neuron's language-wise probabilities of being active, where a neuron is considered active when its activation value is positive. However, this approach may not fully capture the multilingual nature of mLLMs, where language representations are distributional and mutually related. We propose Distribution-aware Language Neuron selection, which leverages pairwise relationships between per-language activation distributions over the full activation range, including negative values. Specifically, we quantify each neuron's language specificity by clustering languages using pairwise overlap coefficients between their activation distributions. Across two mLLMs and two held-out corpora, our identifier more effectively isolates language-specific causal effects, yielding up to 4.9$\times$ higher on-target language damage per neuron while preserving off-target language performance.