Large language models benchmarked for realistic human psychological behavior
Benchmarking Psychological Dynamics in Generative Agents
Human-Computer InteractionArtificial Intelligence
Summary
It is hard to tell how well computer programs that imitate people capture real human thoughts and feelings over time. The authors created a test to measure how closely these programs match human personality traits and behaviors as people experience events. They checked many language models and found most can guess general personality directions but struggle with detailed, moment-to-moment changes. One model balanced the two best. This test helps improve computers that act like real people, especially for studying cause and effect.
What this means in practice
- •For marketing analysts: Use computational personas that reflect psychological traits accurately over time to test customer behavior predictions and campaign effects.
- •For healthcare data teams: Deploy realistic agent models to simulate patient psychological responses for evaluating interventions without risking patient safety.
Authors
Sumer S. Vaid, Ashley V. Whillans
Abstract
Large language models (LLMs) are increasingly deployed to simulate human behavior, acting as computational replicas of human subjects. Yet the lived psychological experience of humans is difficult to benchmark, particularly as it unfolds over time. We introduce a psychometric benchmark for computational replicas: personas that carry a fixed identity through an evolving sequence of events. Built entirely from published norms and meta-analytic effects, the benchmark scores two dimensions of psychological realism. The first, internal validity, quantifies whether generated trajectories reproduce the internal structure of repeated human measurement: distributions, the between- versus within-person variance partition, temporal dependence, and range. The second, external validity, quantifies whether replicas recover established trait, state, and indicator relations. Across 36 open-weight and proprietary LLMs from nine developers (1B-671B parameters), most recover the direction of established relations (84.3% mean agreement) and the variance partition (26 of 34), yet the within-person correlation and distributional structure elude recovery. Internal validity is independent of scale and capability: a mid-size open LLM (Gemma-3-27B) strikes the best trade-off between the two dimensions. The benchmark is a precondition for using computational replicas in causal inference across domains (e.g., marketing, healthcare), and identifies within-person grounding as the central challenge ahead.