Artificial societies benchmark tests how synthetic data matches real people
Artificial Societies Benchmark: A Validation Framework for Synthetic Research
Computation and Language
Summary
Sometimes computer-generated groups of people give average answers that look right but hide important differences and relationships between people. The authors created a new way to check if these fake groups behave like real ones across many tests. They found that synthetic data often misses how people really vary or connect their traits, and giving more details about people helps some models but confuses others. Their test scores help researchers know what parts of the synthetic data they can trust and where real human data is still needed.
What this means in practice
- •For social simulation teams: Check if synthetic social data realistically reflects human variation to improve behavior models.
- •For market research analysts: Assess whether artificial consumer profiles can support reliable analysis of survey responses.
Authors
Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis, James K. He
Abstract
A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions. We introduce the Artificial Societies Benchmark to help researchers assess whether synthetic populations support their intended analyses. The framework combines eleven tests across internal, construct, and external validity, drawing on twenty human sources and comparing nine language models. It connects each research use to the evidence it requires and tests how results change with the information we supply about respondents. Importantly, strong performance in one domain does not establish fidelity in the others. Models often answer too consistently, compress response scales, and alter relationships between traits whilst richer profiles improve prediction for some models and worsen it for others. The resulting scorecard helps researchers identify which aspects of a synthetic population can support their analysis and where researchers need further human evidence.