Large language models struggle to track culture changes over time
ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models
Computation and LanguageComputers and Society
Summary
Language models that read and write text often guess how people think in different countries. However, these guesses don’t change much even when people’s real opinions do. The authors created a set of questions asked in many countries over many years to test this. They found that models usually miss how attitudes evolve and act like culture is the same over time. This suggests language models need new ways to understand how culture changes.
What this means in practice
- •For chatbot developers: Improve chatbot responses by testing if models reflect changing cultural attitudes over time and adjusting training or prompts accordingly.
- •For social media content teams: Assess if AI-generated content aligns with current cultural attitudes rather than outdated stereotypes across different countries.
Authors
Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang
Abstract
Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute of a country or region. We introduce ChronoCulture, a benchmark of 10,360 temporal pairs built from longitudinal survey data across 72 countries. Each pair contains the same survey question asked in the same country in an older and a newer year, with full population response distributions as ground truth. Across six LLMs, we find that models can often predict a country's attitudes at one point in time but do not reliably capture how those attitudes change. Their predictions for different years tend to stay close together, failing to capture much of the observed change. Specifying the intended year in the prompt also offers little overall improvement in alignment with survey responses. These findings suggest that models often present a largely static picture of cultural attitudes, highlighting the need to evaluate cultural awareness across time.