Multilingual text embeddings improved with paired residual removal method
Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal
Computation and Language
Summary
Generating good text representations in many languages is challenging because similar sentences in different languages do not line up well in the model’s understanding. The authors studied how language-specific differences affect these representations and developed a way to reduce these differences without retraining the models. Their method compares pairs of sentences that mean the same thing in different languages and removes the parts that are unique to each language, helping the model focus on shared meaning. This approach improved performance on tasks involving many languages without needing extra training.
What this means in practice
- •For machine translation engineers: Improve cross-lingual sentence matching by reducing language-specific noise in embeddings without retraining models.
- •For multilingual search developers: Enhance search accuracy across languages by aligning semantic meanings better in embeddings from large language models.
Authors
Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi, Manabu Okumura
Abstract
While deriving text embeddings directly from frozen LLMs has emerged as a promising direction, existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. In this work, we present the first systematic study of multilingual training-free text embeddings in LLMs. Our analysis reveals a language-conditioned embedding geometry in which semantically equivalent sentences remain misaligned across languages. We hypothesize that language-specific variation can be disentangled from semantic content to improve embedding quality. To this end, we propose Paired-Residual Subspace Removal (PRSR), a training-free inference-time intervention that steers LLM representations toward a more language-invariant semantic space. PRSR first constructs paired residuals from multilingual parallel sentences to capture language-specific deviations. It then estimates layer-wise low-rank subspaces from these residuals and suppresses the corresponding components in the last-token hidden states. We conduct more than 200 monolingual evaluations across 18 languages using MTEB and MMTEB, along with cross-lingual bitext mining between English and over 200 other languages. Empirical results demonstrate that PRSR consistently improves the performance of existing training-free baselines in both monolingual and cross-lingual settings across different LLMs.