Llm digital twins can mimic human averages but not individual differences
When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability
Artificial Intelligence
Summary
Collecting data from people again and again can be slow and costly, so scientists want to see if AI models called digital twins can stand in for humans in studies. The authors found that these AI twins can predict average human behaviors well but struggle to accurately mimic how individual people differ. Improving the models or adding more personal info helps some, but it doesn’t reliably cut down the need for real human data. The key lesson is that just copying overall human results isn’t enough—AI must support trustworthy scientific conclusions to truly replace humans in measurement.
What this means in practice
- •For behavioral research teams: Assess whether AI-generated responses can reduce the need for repeated human data collection in behavioral experiments while maintaining valid conclusions.
- •For survey design teams: Determine how digital twins can or cannot substitute for human respondents to improve survey efficiency without compromising data quality.
Authors
Steven Wang, Kyle Hunt, Shaojie Tang, Kenneth Joseph
Abstract
LLM-based digital twins promise to reduce repeated human data collection by generating person- specific responses, yet existing evaluations provide little evidence about whether they can reduce human measurement while preserving valid inference. To address this, we introduce statistical substitutability, an inferential criterion that evaluates the extent to which twin predictions can reduce human measurement for a particular estimand while preserving valid inference. We develop a framework, grounded in mixed-subject and prediction-powered inference, that evaluates statistical substitutability along four dimensions: aggregate fidelity, paired respondent-level signal, finite-sample human-label recovery, and stability across populations. Across two empirical evaluations spanning behavioral experiments, multiple models, and alternative respondent representations, we find that digital twins can reproduce average human effects while providing little information about which individuals differ from those averages. Newer models and richer respondent information improve some dimensions of performance but do not reliably translate into human-data savings. Human calibration can reduce aggregate prediction error, yet limited labeled samples often fail to produce stable precision gains. Importantly, these findings demonstrate that behavioral fidelity is neither necessary nor sufficient for statistical substitutability. More broadly, they suggest that AI-generated evidence should be evaluated based on its ability to support valid scientific inference rather than its ability to reproduce human outcomes alone. Digital twins should therefore be judged for confirmatory use by whether they reduce uncertainty about human quantities, not merely by whether they reproduce human means, distributions, or effects.