Discussion among AI agents helps less when disagreement is hidden
Multi-agent discussion gains less when dissent is withheld
Computation and LanguageMultiagent Systems
Summary
This paper studies how groups of AI language models discuss and make decisions together. The authors found that when some agents hide their disagreements, the group benefits less from discussion. They created a simple model explaining that if agents often withhold dissent, discussion is less likely to correct wrong majority opinions. By encouraging agents to share disagreements openly, the group’s overall accuracy improves.
What this means in practice
- •For ai system developers: Improve consensus accuracy in multi-agent AI teams by designing protocols that reduce withholding of dissent.
- •For customer support platform builders: Enhance automated response quality by ensuring disagreement among AI agents is openly considered during collaborative decision making.
Authors
Chand Sahil Mansuri, Xin Wang, Mengying Li, Bryan Acton, Rory Eckardt, Dhaval Patel, Sadamori Kojaku
Abstract
Multi-agent systems of LLMs add discussion to majority voting and are therefore expected to be more capable. However, empirical reports conflict on whether discussion improves accuracy or leads to an incorrect consensus. Here, we introduce a parsimonious model that explains when discussion improves accuracy and when it ends in an incorrect consensus, built from four behaviors repeatedly observed in LLM agents: (1) withholding dissent, (2) internalizing a stated answer, (3) reconsidering after seeing dissent, and (4) correcting toward the correct answer. The model shows that discussion can overturn an incorrect initial majority only when the withholding rate $c$ is below a critical rate $c^* = γ/(γ+ a)$, set by the net correction rate $γ$ and the internalization rate $a$. We estimate these rates from conversation logs with a Bayesian method and place LLM teams relative to $c^*$. As the model predicts, the gain from discussion shrinks as withholding rises, across LLMs and on a hidden profile benchmark, HiddenBench, and MedEInst. Instructing agents not to withhold dissent increases this gain. Turning reasoning off also increases the gain, because reasoning raises the internalization rate $a$ and keeps agents from reconsidering a minority answer. These findings reconcile the conflicting reports and identify when discussion outperforms majority voting.