Multilingual language models struggle with consistent factual error rejection
SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection
Computation and Language
Summary
Large language models (LLMs) perform well in many languages but are usually tested only on picking correct answers, not really understanding facts. The authors created SWORD, a new test that checks if models can spot wrong facts in eight languages by changing Wikidata facts in tricky ways. They found that models often rely on familiar word patterns instead of true fact-checking, sometimes doing worse with Asian languages when facts are distorted. This shows that even if models seem good overall, they can have hidden weaknesses depending on the language and type of mistake.
What this means in practice
- •For multilingual ai developers: Improve testing of language models to detect inconsistencies in fact verification across multiple languages using distortion-based benchmarks.
- •For machine translation teams: Identify and address language-specific biases in factual reasoning when translating or generating content in East Asian languages.
Authors
Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun
Abstract
Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding. We introduce Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark that evaluates whether models consistently reject factual errors across languages. SWORD generates syntactically well-formed but factually incorrect statements in eight widely spoken languages through controlled perturbations of Wikidata triples, ranging from random entity substitutions to semantically plausible property-based selections. Our distortion-based evaluation surfaces two critical insights that remain entirely obscured by conventional benchmarks. First, models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical random substitutions, suggesting reliance on distributional familiarity rather than genuine factual verification. Second, models exhibiting comparable baseline accuracy across languages show substantial performance degradation specifically on (East) Asian languages when presented with distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points (49\% relative reduction) in some models. These findings demonstrate that multilingual factual reasoning involves asymmetric capabilities that aggregate accuracy metrics systematically obscure.