Silicon sampling predicts country views but not individual opinions

Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey

Computers and Society

Summary

The study looks at how well large language models (LLMs) can predict survey results from different countries by pretending to be survey respondents. The researchers found that LLMs can moderately estimate average country-level answers when given the country's name but cannot accurately predict individual people's answers. The way answers are formatted also affects the model's predictions. In fact, just averaging neighbor countries’ real answers works better than the models for country-level estimates. This means these models are useful for rough country comparisons, but not for understanding individual attitudes or full answer distributions.

What this means in practice

  • For social survey analysts: Generate preliminary country-level survey comparisons when actual individual data is unavailable by simulating responses with country context.
  • For market research teams: Use LLMs to roughly estimate country-level consumer attitudes for exploratory analysis, understanding that individual precision is poor.

Authors

Chuyao Wang

Abstract

Silicon sampling uses large language models (LLMs) to simulate survey respondents. Whether it recovers cross-national variation, and why, remains unresolved. This study evaluates it against European Social Survey Round 11 (30 countries, 42 items) with two open-weight LLMs under first- and third-person prompts, plus backstory and response-format experiments. Aggregate recovery is moderate and uneven across items. Adding the country name to a three-variable demographic backstory raises the median per-item correlation between simulated and observed country means from -0.03 to 0.52, and the richer profiles tested add no consistent gain. The respondent's country label acts as a country-level assumption that respondent detail does not revise. Naming the response-scale endpoints in words stops the model from ranking countries backwards, so the answer format sets the direction of the ranking. Individual-level recovery remains negligible in every condition and does not track aggregate recovery across countries. An average of neighboring countries, using no LLM, recovers country levels more accurately than every model condition and ranks them about as well. Silicon sampling can thus support exploratory country-ranking comparison after item-level validation and with the response format reported. It does not support individual or distributional inference.
View PDFOpen arXiv