Model refusal varies when language components are translated differently
Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation
Computation and LanguageCryptography and Security
Summary
When computer models respond to requests in different languages, the authors found it's important to see which part of the request is translated. They tested how changing the language of either the request's components or the operation affected model safety, like refusing unsafe requests. Some models showed small differences, but one model refused harmful requests more often when parts were translated to Spanish. The work suggests checking refusal and unsafe compliance separately while tracking which component's language was changed.
What this means in practice
- •For multilingual ai developers: Detect shifts in refusal rates when translating different parts of input to improve safety in multilingual AI systems.
- •For ai safety teams: Use component-level translation tests to better understand and measure unsafe compliance across languages in AI models.
Authors
Yongxi Zhou, Junwei Yao, Wenbo Ye, Jiahui Ge
Abstract
Whole-prompt translation evaluates a request in different languages but does not compare which component carries a language. We introduce a paired reversal: translate the complete request component or the complete operation component into the same target language, holding behaviour, English frame and translated-span count fixed. Component identity and non-English text quantity change together. Assigning English, Chinese and Spanish independently to the two components yields nine configurations on 190 of 300 sampled harmful behaviours retained after consensus annotation and automatic gating. Refusal scored by language-model judges is the primary outcome; unsafe compliance is reported separately. On Gemini 2.5 Flash and DeepSeek, three refusal swaps are about 5.3 percentage points and the fourth is -0.5; none survives the pre-specified six-test Holm correction. In an exploratory extension, Mistral Small 4 has a 13.7-point Spanish swap (95% interval [8.4, 18.9]), which survives the twelve-test Holm sensitivity correction under the primary judge, and 12.1 points under a second judge. GPT-5.4-mini estimates are within 2 points of zero. Component assignment supplies an additional evaluation dimension. Evaluations should record component languages and translated text quantity, comparing matched behaviours at equal translated-span count while interpreting refusal and unsafe compliance separately.