Large language models struggle to mimic public opinion in debates
From Simulated Citizens to Simulated Deliberation: Challenges in Representation and Interaction
Artificial Intelligence
Summary
Simulating public discussions with AI requires agents that represent real people's opinions and influence each other through interaction. The authors tested whether AI personas based on Korean census data could debate real policy questions like humans do. They found these AI personas don't reliably reflect population opinion patterns and can even reverse demographic trends. While the AI debates produce good arguments and change opinions, much of this change happens without genuine interaction. The study shows that simulating population diversity, generating arguments, and producing opinion shifts through interaction are separate challenges.
What this means in practice
- •For policy simulation teams: Generate reasoned arguments on both sides of policy questions to inform policy analysis without assuming full population representation.
- •For customer support trainers: Use AI personas that produce varied and reciprocal arguments to train conversational agents in handling complex, dynamic exchanges.
Authors
Chaemin Jang, Junsik Min, Jaewoo Choi, Donggyu Lee, Haiin Lee, Junyoung Park, Namhee Kim, Hyunwoo Kim, Jungwon Kim, Juho Kim, Nuri Kim, Jihee Kim
Abstract
Multi-agent LLM deliberation has been explored as a scalable way to simulate public deliberation. For such simulations to be informative, persona agents should reflect population opinion patterns and interaction should shape their conclusions. We evaluate whether LLM-based deliberation can meet these two conditions using census-grounded Korean personas debating real policy questions benchmarked against national surveys. Persona agents do not reliably reproduce population opinion patterns: responses are often far more concentrated and frequently reverse demographic differences in the human data. Deliberations nonetheless produce reasoned, reciprocal, and varied arguments alongside substantial stance movement. Yet much of this movement does not require peer exchange: sealed-monologue agents change position at similar rates and reach nearly the same final balance as full debates, while groups initialized with very different positions often converge to similar endpoints. Anchoring population-informed starting positions, meanwhile, sharply suppresses updating. Thus, population representation, argument generation, and interaction-driven opinion change do not necessarily go together. The simulations readily surface arguments on both sides, though whether they capture the diversity of human perspectives remains untested, leaving open a promising role for argument surfacing even as population simulation requires further validation.