How language shapes answers in cross-language question answering
Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities
Computation and LanguageArtificial Intelligence
Summary
This paper looks at how large language models answer questions when information about a fact exists in one language but is missing or different in another. The authors created a new dataset called Waldo using Wikipedia entries in multiple languages to study this problem. They found that when the same fact is written differently across languages, models tend to give answers based mainly on the language the question is asked in, even if it conflicts with other sources. They also tested ways to reduce this language bias and made progress in balancing answers across languages.
What this means in practice
- •For multilingual chatbot developers: Improve chatbots to provide consistent answers across languages by accounting for language-specific knowledge differences.
- •For content aggregation platforms: Design systems to detect and manage conflicting information from multilingual sources, improving reliability in global knowledge services.
Authors
Dayeon Ki, Ruochen Zhang, Silviu Cucerzan, Ryen W. White, Ning Gao
Abstract
Large Language Models increasingly serve as interfaces for knowledge-intensive information seeking tasks across languages by synthesizing multilingual evidence. Prior work has shown that they often exhibit query-language preference -- the tendency to favor sources written in the language of the query -- but has largely examined this behavior in settings where equivalent knowledge is available across languages. However, this bias becomes consequential when sources in different languages provide incomplete or inconsistent accounts of the same fact, since the information users receive then depends on the sources a model selects to use. To characterize query-language preference under such cross-lingual knowledge disparities, we introduce Waldo, a multilingual Question-Answering (QA) benchmark constructed from Wikipedia. Waldo contains 12K QA pairs targeting knowledge gaps, where a fact is available in one language but absent in another, and knowledge conflicts, where language editions provide conflicting versions of the same fact. Evaluating eight models across five languages, we find that when one language edition merely lacks the relevant fact, models generally use evidence from the other language regardless of the query language. Under conflicting accounts, however, model responses strongly align with the document in the query language, causing semantically equivalent queries to elicit different accounts depending on the user's language. Finally, we explore two different approaches that could mitigate this preference under knowledge conflicts: a mechanistic intervention that ablates attention heads associated with query-language preference, and LoRA-based training, which reduces the preference gap by up to 61.5%.