Language models struggle with conflicting cultural norms in answers
The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs
Computation and Language
Summary
Language models that answer questions using cultural information often default to English-speaking cultural norms when no country is specified. The authors created a test called CultureLens with many examples from different countries to study this. They found that when the model is told a cultural norm that conflicts with the true norm, its answers become less accurate. However, fine-tuning the model with examples of conflicting norms helps it answer correctly most of the time, even for countries it hasn’t seen before.
What this means in practice
- •For multilingual chatbot developers: Improve chatbot cultural sensitivity by training models to handle conflicting cultural norms accurately.
- •For content moderation teams: Use insights to detect and correct cultural norm errors in AI-generated content that might cause misunderstandings.
Authors
Samee Arif, Furkan Danisman, Angana Borah, Honglak Lee, Dehan Kong, Lajanugen Logeswaran, Moontae Lee, Zhijing Jin, Rada Mihalcea
Abstract
Large Language Models (LLMs) increasingly operate across culturally diverse settings, yet existing evaluations largely measure cultural behavior at the output level. We introduce CultureLens, a reference-grounded benchmark and mechanistic study of how LLMs use cultural information. CultureLens contains 9,261 scenarios across 75 countries and systematically varies country, value, true-norm, and contradictory-norm evidence. Across Llama-3.1-8B, Gemma-3-12B, and Qwen-3.5-27B, stating a contradictory norm lowers aligned answers by 32-47 percentage points, even when the model answers correctly from the country alone. Mechanistically, models default to Anglophone norms when no country is given. The failure under a contradictory norm is not one of missing knowledge, as blocking the norm inside the model restores the correct answer in most cases. Models accept a stated norm according to how it is presented and not whether it is true, which leaves them most exposed to errors in retrieved content. Supervised fine-tuning with contradictory-norm examples raises aligned answers under a contradictory norm to 85-93%, including on unseen countries, at a small cost under a true norm.