Large language models partially mimic survey responses after health intervention
How Well Do LLMs Simulate Survey Responses Following a Breast Cancer Screening Intervention?
Social and Information Networks
Summary
Collecting survey answers from people can be hard and tricky because of privacy. The authors studied if large language models (LLMs) could predict how groups of women would answer surveys before and after a breast cancer screening program. They found that LLMs that knew some background info did better than guessing, but they still weren’t as accurate as real participants. The models had more trouble with older women, certain living situations, and questions about cancer beliefs or genetics. This suggests LLMs can help simulate some responses, but they don’t fully replace real data yet.
What this means in practice
- •For public health analysts: Use LLM agents to estimate population survey responses after health interventions when direct survey data are limited or private.
- •For marketing teams: Generate synthetic consumer attitude surveys based on demographic profiles to forecast changes after product campaigns.
Tested on one dataset.
Authors
Kenneth Koh, Ryan Jak Yang Lim, Alessandro Sparacio, Peh Joo Ho, Mile Sikic, Borame L Dickens, Mikael Hartman, Jingmei Li
Abstract
Collecting survey data is laborious and limited by privacy constraints. Large language models (LLMs) have shown promise as predictive social simulations. It is unclear whether they can replicate population-level response distributions before and after a healthcare intervention. Using information derived from 4125 women aged 35-59 years, we evaluate whether agents informed solely by pre-intervention profile information can reproduce post-intervention response distributions. Groups of LLM agents (n=50) were created with Gemma 4 E4B and Qwen3.5 9B; conditions ranged from zero-shot prompting to agent profiles enriched with aggregate or individual-level demographic characteristics and pre-intervention questionnaire responses. We compared predicted and observed response distributions with Total Variation Distance (TVD) and Normalized Wasserstein Distance (NWD). Across both LLMs, profile-based agents improved distributional accuracy relative to zero-shot and random baselines. Nevertheless, direct sampling of 50 real participants remained more accurate. Prediction errors were also higher among participants aged 55-59 years and those living in private property. Errors also varied by question theme and LLM model, with the highest errors observed for cancer fatalism and post intervention attitudes toward genetics. Sensitivity analyses showed that performance was influenced by prompt template changes and temperature hyperparameter. Our results show the potential of LLM-based agents to model behavioral responses to interventions in silico. However, profiles containing additional information beyond demographics did not consistently outperform simpler ones. Certain cultural constructs and population groups also remain inadequately represented by the LLM models evaluated. Future work may include building behaviorally grounded and locally validated virtual populations.