Summary
Collecting survey answers from many people can be slow and privacy is tricky. This study checked if large language models (LLMs), which are advanced computer programs that understand and generate text, can act like groups of people answering surveys about breast cancer screening. The researchers found that these models do better when given some background information but are still not as accurate as asking real people. The models struggled more with certain age groups and types of questions, especially about fears of cancer and genetics. This work shows promise but also highlights that current models don’t fully capture how all groups think and respond.
Large language modelsSurvey simulationBreast cancer screeningBehavioral responsePopulation demographicsTotal variation distanceWasserstein distancePrompt engineeringHealthcare interventionSocial simulation
Authors
Kenneth Koh, Ryan Jak Yang Lim, Alessandro Sparacio, Peh Joo Ho, Mile Sikic, Borame L Dickens, Mikael Hartman, Jingmei Li
Abstract
Collecting survey data is laborious and limited by privacy constraints. Large language models (LLMs) have shown promise as predictive social simulations. It is unclear whether they can replicate population-level response distributions before and after a healthcare intervention. Using information derived from 4125 women aged 35-59 years, we evaluate whether agents informed solely by pre-intervention profile information can reproduce post-intervention response distributions. Groups of LLM agents (n=50) were created with Gemma 4 E4B and Qwen3.5 9B; conditions ranged from zero-shot prompting to agent profiles enriched with aggregate or individual-level demographic characteristics and pre-intervention questionnaire responses. We compared predicted and observed response distributions with Total Variation Distance (TVD) and Normalized Wasserstein Distance (NWD). Across both LLMs, profile-based agents improved distributional accuracy relative to zero-shot and random baselines. Nevertheless, direct sampling of 50 real participants remained more accurate. Prediction errors were also higher among participants aged 55-59 years and those living in private property. Errors also varied by question theme and LLM model, with the highest errors observed for cancer fatalism and post intervention attitudes toward genetics. Sensitivity analyses showed that performance was influenced by prompt template changes and temperature hyperparameter. Our results show the potential of LLM-based agents to model behavioral responses to interventions in silico. However, profiles containing additional information beyond demographics did not consistently outperform simpler ones. Certain cultural constructs and population groups also remain inadequately represented by the LLM models evaluated. Future work may include building behaviorally grounded and locally validated virtual populations.