Large language models underestimate cultural differences in everyday norms
Large language models underestimate and partly misrepresent cultural variation in everyday norms
Computers and Society
Summary
How well can large language models like GPT know the normal everyday behaviors in different cultures? The authors tested four popular models by comparing their guesses to human survey data from 90 societies about everyday social rules. The models often missed how much these behaviors vary between cultures and sometimes got the pattern of which societies found behaviors acceptable wrong. The models did better with some topics, like kissing or flirting, and when prompted in local languages, but still underestimated cultural differences overall.
What this means in practice
- •For multilingual chatbot developers: Improve chatbot responses by understanding that current large language models poorly capture cultural differences in social norms across societies.
- •For social ai integration teams: Use insights about local language prompting to better tailor AI outputs respecting cultural norms in various countries.
Authors
Kimmo Eriksson, Irina Vartanova, Pontus Strimling
Abstract
A key aspect of culture is a society's norms about everyday behavior. How accurately do large language models (LLMs) represent cultural differences in such norms? To answer this question we used the recent Global Study of Everyday Norms (GSEN), which collected ratings of 150 scenarios in 90 societies, as the human benchmark. We prompted GPT-5 to estimate each society's average rating for every scenario, and later repeated the benchmark in three other LLMs: GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro. Compared to GSEN estimates, all four LLMs misrepresented cultural variation in two ways. First, they greatly underestimated its magnitude, estimating differences between societies to be, on average, less than half their measured size. Second, for many scenarios the LLMs poorly identified the pattern of variation, that is, which societies judged the behavior less acceptable and which societies judged it more acceptable. The pattern of variation was identified better for scenarios that elicit concerns about vulgarity, especially scenarios involving kissing and flirting. We also found that norms in more developed societies tended to be estimated somewhat more accurately, and that prompting in local survey languages rather than English produced only a modest improvement in accuracy. Local-language prompting also reduced, but did not remove, the underestimation of between-society differences. Cultural differences in everyday norms are only weakly and unevenly represented by LLMs.