Multilingual translation errors revealed by opposite premise tests
When Opposite Premises Translate Identically: A Source-Conditioned Diagnostic for Multilingual Commonsense Items
Computation and LanguageArtificial Intelligence
Summary
When translating questions or statements from English to other languages, it’s important that the meaning stays the same. The authors found that sometimes when you flip a statement to its opposite meaning but translate both versions, the translations end up identical, which shows a problem. This means some translations don’t keep the intended meaning accurately. They tested this with several languages and translations but didn't identify exactly which part went wrong or how it affects answers.
What this means in practice
- •For machine translation developers: Detect translation errors by testing if opposite premises yield identical translated outputs to improve model fidelity.
- •For language model evaluation teams: Use source-conditioned diagnostics to identify unfaithful translations that can affect multilingual reasoning benchmarks.
Authors
Shuai Wang, Rui Hu
Abstract
Multilingual reasoning comparisons depend on translated premises preserving the relations in their English sources. Automatic translation scores can rank source-target pairs, but an acceptance decision alone cannot certify a counterfactual benchmark field. We test a source-contrast diagnostic for ACCORD premises: reverse the English polarity, translate both arms independently, count identical target outputs, and score each arm against its own source. Across 71 matched question identities at two containing-item depths, five target languages, and three translation checkpoints, Swahili has 63, 113, and 67 exact collisions among 710 pairs per checkpoint. At a human-reference-calibrated threshold, round-trip NLI passes both arms of 3 of the 113 NLLB-3.3B collisions, while LaBSE passes 78. Each collision proves at least one unfaithful translation, but does not identify its arm. These controlled measurements do not establish screening accuracy on spontaneous translation errors or a language-by-hop reasoning effect; the study has no native equivalence labels or answer-model evaluation.