Translation circuits improve low resource multilingual language models
Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation
Computation and Language
Summary
Translating between languages with few examples is really hard for AI models. The authors study how these models translate between Hindi and related languages by looking inside the model’s 'circuits'—the important parts that do the actual translating. They made a method called TITAN that tweaks only these key parts instead of the whole model, making translation better without lots of extra work. Their approach works much better than usual methods, especially for less common language pairs, and helps explain how the model handles languages without relying on English.
What this means in practice
- •For machine translation engineers: Improve low-resource language translation models efficiently by fine-tuning only crucial transformer circuits identified by TITAN.
- •For natural language processing platform developers: Integrate targeted circuit optimization to boost translation quality in multilingual AI services for languages with limited data.$Commercial implications: Enables better multilingual translation products for underserved languages, giving a commercial edge in emerging market applications.
Authors
Atai Waris Khan, Abdullah Mazhar, Raghu Kumar, Md Shad Akhtar
Abstract
Recent multilingual large language models achieve strong translation performance, yet the internal mechanisms responsible for low-resource multilingual translation remain insufficiently understood. In this paper, we study translation circuits in multilingual decoder-only LLMs for Hindi-to-Urdu and Hindi-to-Sindhi (Devanagari). We propose TITAN, a mechanistic framework that combines causal intervention, sparse feature decomposition, and targeted preference optimization to identify and optimize crucial translation components in the transformer. Unlike conventional full-parameter fine-tuning, TITAN restricts optimization to sparse, causally important circuits, enabling scalable and parameter-efficient adaptation while updating fewer than 5% of model parameters. Experimental results show that TITAN achieves BLEU improvements of 33.75% on Hindi-to-Urdu and 45.86% on Hindi-to-Sindhi over the strongest multilingual baselines. Human expert evaluation further confirms substantial gains in adequacy and fluency across both language pairs. Additional analyses reveal that multilingual translation is implemented through sparse, structured circuits operating over a shared semantic subspace rather than through an English-centric pivot representation. Overall, our findings demonstrate that mechanistic circuit discovery can directly improve low-resource multilingual translation while providing interpretable insights into multilingual transformer behavior.