Multilingual evaluation method reveals language bugs in large language models
Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
Computation and LanguageArtificial Intelligence
Summary
Current large language models (LLMs) are mostly tested for safety and reasoning in English, but they are used worldwide in many languages. The authors developed a new way to translate test problems into multiple languages without changing their meaning, especially for tricky cases like harmful questions or math puzzles. They created tests in 11 languages and found that LLMs can behave inconsistently or worse when dealing with less common languages. Their work helps better understand how these models perform beyond English and highlights the need for careful multilingual testing.
What this means in practice
- •For language technology developers: Test multilingual AI systems consistently by using a new set of aligned safety and reasoning evaluation benchmarks across 11 languages.
- •For content moderation teams: Improve harmful content detection by using multilingual evaluation tools that reveal language-specific refusal and response inconsistencies in large language models.
Authors
Zahraa Al Sahili, Ioannis Patras, Matthew Purver
Abstract
Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94\% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.