Language models overstate agreement compared to human groups in reasoning tasks
Language-model groups overstate consensus when replaying human deliberation on a reasoning task
Artificial IntelligenceComputation and LanguageComputers and SocietyMultiagent Systems
Summary
When groups of people think and discuss a tricky puzzle, their answers don’t always end up as the same. The authors compared these human groups with groups made up of computer language models acting like the people. They found that these model groups agree with each other much more than real people do, often agreeing on wrong answers. This means language models might make group thinking look more unanimous than it really is. Their work shows it’s important to check how we measure agreement when simulating human group decisions.
What this means in practice
- •For ai safety engineers: Assess and calibrate group agreement in AI simulations to avoid overestimating consensus during decision making.
- •For human-computer interaction designers: Design group collaboration tools that better reflect realistic levels of agreement based on this model-human comparison.
Authors
Tengfei Shao
Abstract
Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These complementary routes reduced different measurement asymmetries yet converged within 0.5 percentage points. The gap persisted without early stopping and under a reparameterization removing the memorizable answer; reasoning-mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief-anchored agent groups were biased estimators of the human group-outcome distribution in this setting. These analyses provide a scoring-explicit basis for assessing simulated-group estimates of human deliberative outcomes.