Large language models fail to preserve cultural differences in surveys
Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populations
Computation and LanguageArtificial Intelligence
Summary
When large language models (LLMs) are used to simulate answers to surveys from people in different countries, they often fail to keep the unique cultural differences between those countries. The authors found that common ways to measure how well these models replicate real survey data miss this problem. They created a new way to check if the models flatten or exaggerate cultural differences. Their tests showed this new measure catches issues that standard tests miss, helping spot when survey simulations are too similar across cultures or overstated.
What this means in practice
- •For survey data analysts: Improve evaluation of synthetic survey data by detecting loss or exaggeration of cultural differences in model-generated responses.
- •For language model developers: Develop and test prompting methods that maintain distinct cultural patterns in responses across countries for better cross-cultural fidelity.
Authors
Yeeun Chae, Yewon Choi, Seunghyun Lee, IL Im
Abstract
Large language models (LLMs) are increasingly used as synthetic survey respondents to estimate population response distributions. In cross-cultural survey simulation, evaluations should assess not only distributional fidelity within countries but also whether differences across countries are preserved. However, existing distance-based metrics such as Jensen--Shannon divergence (JSD) do not directly capture such cross-country differences. To address this limitation, we introduce Cultural Divergence Preservation (CDP), a reference-light diagnostic based on a one-time human calibration. CDP identifies reduced cross-country divergence as cultural flattening and increased divergence as cultural caricature. To evaluate CDP, we conduct experiments across four LLM backbones, three persona-based prompting methods, and two survey domains, the World Values Survey (WVS) and the Big Five Personality Test. The results reveal a systematic discrepancy between conventional fidelity metrics and CDP. Controlled experiments show that CDP changes monotonically as cross-country divergence is attenuated or amplified, while the corresponding changes in JSD remain relatively small. In our audit of real LLM generations, DeepPersona-Inspired prompting is frequently favored by conventional fidelity metrics but exhibits the strongest flattening in every model--domain block. CDP thus complements fidelity metrics by directly quantifying the attenuation or amplification of cross-country divergence.