Multilingual language models use separate circuits for multi-step reasoning

Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs

Computation and Language

Summary

Multilingual large language models can answer complex questions that require multiple steps in different languages, but it's unclear if they use the same internal mechanisms for all languages. The authors identified specific parts of these models, called Bridge Routing Heads, that handle this reasoning separately for each language. Changing these parts can greatly harm or improve the model’s ability to answer questions in a particular language. They found two different languages use mostly separate 'circuits' inside the models, revealing how reasoning is organized internally. This work shows it’s possible to fix language-specific reasoning errors by intervening directly on these internal components without retraining the whole model.

What this means in practice

  • For multilingual ai developers: Improve cross-lingual reasoning in large language models by modulating identified internal routing heads without retraining.
  • For ai system reliability teams: Diagnose and fix language-specific reasoning failures in deployed multilingual language models via targeted activation intervention.

Authors

Seunghan Kim, Minyeong Choe, Hyunil Kim, Haehyun Cho

Abstract

Multilingual LLMs answer the same multi-hop reasoning question across languages, but we lack a mechanistic account of whether they share an internal circuit. We identify Bridge Routing Heads (BRH) in two large multilingual LLMs through a three-stage pipeline. The resulting language-specific head sets exhibit near-complete mutual exclusivity across the five languages, with a mean Jaccard similarity of only 0.017 for Llama 3.1 70B and 0.057 for Qwen 2.5 72B, revealing language-idiosyncratic circuits. Ablating general BRH increases two-hop Negative Log-Likelihood (NLL) by 39-89x the random-head baseline, providing direct causal evidence of their role. Amplifying these heads in a failing target-language pass rescues up to 51.7% of cross-lingual failures, with no training. The two models share this dual-circuit pattern but allocate heads differently: Llama concentrates chaining in a large general pool, while Qwen leans on larger language-specific pools. Together these results show that activation-level intervention alone can recover correct answers from cross-lingual reasoning failures.
View PDFOpen arXiv