Validating synthetic people to better predict real behaviors
Consequential Behaviour and Representational Fairness in the Validation of Synthetic Research
Computation and LanguageComputers and Society
Summary
Synthetic surveys use AI-created people to model real human behavior for research and decision-making. The authors point out that current methods often check if these fake people seem like real people, but miss whether they actually behave like them in important ways. They suggest a new way to validate synthetic data, focusing on how well it predicts real actions, and whether it fairly represents different groups. They tested this idea with an example involving electric vehicle charging prices and provide guidelines for others to use.
What this means in practice
- •For policy analysts: Validate synthetic populations used to forecast policy impacts on different demographics with fairness and accuracy.
- •For price strategy teams: Use the validation framework to assess synthetic data predicting customer responses to new pricing models like EV tariffs.
A position paper. It proposes an approach and reports no results.
Authors
Florian Kutzner, Celina Kacperski, Laura de Molière, Edoardo Chidichimo, Min Jun Jung, Felix Patrick Sedgwick Wallis, James Kunling He
Abstract
Researchers in industry and academia use synthetic survey respondents powered by large language models as substitutes for human samples. These synthetic populations require validation against real-world data, so researchers often address them using ad hoc comparisons with human surveys. Inspired by the intention-behaviour gap in behavioural science, we argue that these validations test the wrong thing for most applied cases where decision makers commission synthetic research to anticipate consequential behaviour. To address this problem, we propose a validation framework with two requirements. First, every validity claim must state its level of correspondence with human data: does the sample predict what the represented people do, which of four diagnostics (location, dispersion, response process and structure) does the validation address, and does the validation compare against experimental effects? Second, researchers must report validity claims for subgroups, since these groups are often the most affected by consequential decisions and aggregate accuracy hides their misrepresentation. Our validation framework operationalises three justice dimensions (distributional, procedural, and recognition) as measurable quantities and defines within-persona counterfactual experiments as a validation requirement. We then apply the framework to electric vehicle charging tariffs, before closing with a reporting checklist that researchers can use to make convincing validity claims.