Training with ciphers improves rare term translation in large models
Training with Ciphers to Enhance In-Context Terminology Learning For Translation from Rare Languages
Computation and LanguageMachine Learning
Summary
Large language models can learn to translate new words by using examples shown during deployment, but they struggle with rare or unfamiliar terms. The authors identify this problem and introduce a method called RareCipherFT, which hides rare words in examples by scrambling their letters so the model must copy them to learn accurately. Their approach helps models remember and translate rare words better across several languages without hurting overall translation quality. Tests confirm the improvements come from linking source and target rare terms, not from accidental shortcuts.
What this means in practice
- •For machine translation developers: Improve rare word translation accuracy by integrating RareCipherFT strategies into fine-tuning workflows for language models.
- •For multilingual chatbot teams: Enhance understanding and output of low-resource language terms in conversations by applying RareCipherFT training methods.
Authors
Deevyanshu Malu, Deeptanshu Malu, Sunita Sarawagi
Abstract
Fine-tuning LLMs with in-context examples (IC-Train) lets translation models adapt to new parallel data at deployment, but it is unclear whether they actually learn the translations of rare terms from those examples. We formalize this capability as in-context term learning (ICTL). We show that standard IC-Train develops this capability only weakly, and propose RareCipherFT, which strengthens it without any lexical resources. RareCipherFT pairs each training target with in-context examples that share a rare term with it. It then replaces that term with a character-shuffled cipher in both the context and the reference, so the term can only be recovered by copying from context. The ciphered fraction of training data is annealed to preserve translation quality. Across seven translation directions between English and Tamil, Assamese, Mizo, Nepali and Hausa, and four models from 1B to 7B parameters, RareCipherFT raises term recall over standard IC-Train by 5-15 absolute points without lowering COMET. Probes using synonyms, nonce words and controls for shared surface context show that these gains come from linking source terms to their in-context translations rather than from shortcuts.