Large language models give different answers across languages on debated topics
A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics
Computation and Language
Summary
Many questions don’t have clear right or wrong answers because people around the world see things differently. The authors created a new dataset with questions in 18 languages about topics people often disagree on. They tested many language models and found bigger ones give more consistent answers in different languages, while smaller ones change their answers more depending on the language. They also found that small changes in how a question is asked can affect the model’s opinion, even for the largest models.
What this means in practice
- •For multilingual ai developers: Improve testing of language models across languages by including perspective diversity on controversial topics using the PerspectivesQA dataset and ROOTS framework.
- •For content moderation teams: Identify language and model size impacts on viewpoint consistency to better handle subjective or contested online content moderation decisions across languages.
Authors
Wiem Ben Rim, Yao Lu, Pontus Stenetorp, Emine Yilmaz
Abstract
Large Language Models(LLMs) have shown strong performance on tasks with verifiable answers, but many real-world queries involve subjectivity, disagreement, and conflicting evidence. Most existing benchmarks for such queries remain English-centric, overlooking the vast diversity of perspectives held by users worldwide. To systematically study how LLM perspectives vary across languages, we curate PerspectivesQA, a manually verified multilingual dataset of globally contested topics covering 18 languages, and propose ROOTS, a framework that profiles linguistic, perspective, and framing diversity. We evaluate a diverse set of eight models covering both open-source and proprietary families at varying scales, establishing that LLMs hedge their answers in English but commit to positions in lower-resource languages. Larger models maintain a more consistent cross-lingual worldview, while smaller models are more easily swayed. All models also exhibit high sensitivity to minor surface-level input variations, producing notable stance even in larger models.