Large language models generate survey answers that miss real human response patterns
Silicon Samples Match the Answers but Not the Respondents
Computation and LanguageComputers and Society
Summary
The paper looks at how large language models (LLMs) create fake people to answer survey questions based on given demographics. These models can get individual answers right but often produce answer combinations that no real person actually gave. The authors introduce a new way to measure this problem, showing that LLM-generated respondents differ significantly from real ones. Even simple methods mixing answers independently from real survey data outperform LLMs at matching real human answer patterns.
What this means in practice
- •For survey methodologists: Identify weaknesses in synthetic respondent generation to improve survey data simulation accuracy.
- •For statistical data scientists: Use respondent coverage gap metric to evaluate and select better synthetic population models for social science analysis.
Authors
Yusuke Kumagae, Masahiro Kaneko
Abstract
Silicon sampling generates survey respondents by having a large language model (LLM) answer a survey as a person with given demographic attributes, and the resulting synthetic populations are increasingly used in place of human ones. Existing evaluations focus on individual-answer accuracy, per-question answer distributions, and correlations across questions. None of these compares the respondents themselves, so a generator whose respondents have combinations of answers found in no human respondent can still score well if it reproduces every answer distribution. We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples. On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals. Across the prompting and inference settings we test, every LLM has a higher RCG than a Gaussian copula fitted to the human data.