Multilingual language models give same answers but different explanations
Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models
Artificial IntelligenceComputation and Language
Summary
Large language models that work in many languages often produce the same final answers, but their step-by-step reasoning can differ a lot across languages. The authors found that simply comparing the overall response hides big differences in how these models explain their answers. They created a detailed way to compare reasoning steps and showed humans agree more with this detailed comparison than with overall answer similarity. This finding means we need better tools to check if people using different languages get equally solid explanations from these models.
What this means in practice
- •For multilingual chatbot developers: Ensure chatbots provide consistent and comparable explanations across languages to improve user trust and safety verification.
- •For multilingual content moderation teams: Use trace-level analysis to audit and verify explanation consistency in different languages, reducing hidden biases in moderation decisions.
Authors
Rui Sun, Peizhao Mei, Xiewei Cheng, Kaizhuo Chen, Taoran Li, Zhiyuan Yu, Varun Chandrasekaran
Abstract
Multilingual large language models are increasingly deployed in settings where visible reasoning traces support verification, education, and safety auditing, yet standard evaluations typically measure only final-answer agreement or holistic response similarity. As a result, they establish outcome consistency without determining whether users across languages receive comparable supporting evidence. We introduce a trace-level divergence framework that compares visible reasoning traces as structured sequences, decomposing agreement into structural similarity, order consistency, and holistic similarity, with pair-specific calibration to enable comparison across languages. Across three reasoning benchmarks, seven languages, and six frontier LLMs, the natural baseline---embedding each whole response and comparing the two by cosine---systematically overstates agreement: $52.1\%$--$67.3\%$ of examples in the top quartile of holistic similarity are not in the top quartile of structural similarity. This dissociation is stable across sentence encoders and step segmentations, and blind bilingual annotation studies in two English-anchored language pairs show that human judgments align with structural similarity rather than with the holistic baseline. A same-language control shows that the level of divergence is language-specific: two English traces for paraphrases of the same question agree more closely than an English trace and its counterparts in other languages. Our results show that answer-level and holistic evaluations can substantially overestimate the consistency of multilingual explanatory artifacts, motivating trace-level evaluation for reliable multilingual deployment.