Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how well large language models (LLMs) can act as social simulators by comparing their reasoning to real humans'. They focused on a sunscreen product test where people explained why they liked or disliked products. By turning these explanations into reason states, they tested if LLMs could mimic human reasoning, not just the final choice. They found that while LLMs often produce plausible reasons, these don’t always match the actual human thought process. The authors propose a new way to evaluate simulators based on how well their reasons line up with human evidence.
large language modelssocial simulatorsreasoning patternssunscreen concept testrationale-derived reasonsreason statespurchase intent predictionsynthetic survey respondentsevaluation frameworkhuman-computer comparison
Authors
Atharva Pandey, Gautam Jajoo
Abstract
Large language models are increasingly used as social simulators, including as synthetic survey respondents. Most evaluations ask whether simulated outcomes resemble human outcomes. We argue that this is necessary but too weak: a simulator can match the final answer while using the wrong rationale-derived reason pattern. We study this problem through a 94-person sunscreen concept test in which each respondent evaluated three product concepts and wrote open-ended rationales. We map those rationales into signed reason states $Z$, where positive signs support adoption and negative signs block it. This gives a practical audit: holding respondent descriptors $D$, category context $K$, and concept treatment $X$ fixed, do human rationale-derived reasons help predict behavior $Y$, and can an LLM simulate the same reason state without seeing the human rationale or outcome? Human rationale-derived reasons substantially improve held-out prediction of purchase intent. LLM-simulated reasons are more brittle: they often sound plausible, but frequently echo the concept board rather than recover the respondent's acceptance or rejection path. The paper contributes an evaluation framework for social simulators. Reason states do not identify natural causal effects by themselves, but they provide an interpretable test of whether a simulator's stated reasons align with human evidence.