Multi-agent AI debates change answers but not their quality or certainty
A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate
Artificial Intelligence
Summary
People often think that when several AI agents debate, they will give better answers by showing real disagreement. The authors tested this idea by measuring how much AI agents agree, whether their replies actually challenge each other, and if their opinions stick after the debate instructions change. They found that while the way the agents talk shifts a lot depending on the tone of debate, their final answers do not improve in quality or confidence. In fact, an unbiased judge saw no clear winner or better answer from these debates.
Large Language Models (LLMs)multi-agent debateagreement measurementstance persistencereply text analysisGlobalOpinionQAtone in AI communicationanswer quality evaluationbias in judgingtoken log-probabilities
Authors
Chen Qian
Abstract
Multi-agent debate, in which several LLMs exchange arguments before answering, is widely assumed to improve answer quality by surfacing genuine disagreement. That mechanism is rarely checked. We introduce four measurements: (A) the agreement a debater reports; (B) whether its reply text actually pushes back; (C) whether the position persists once the eliciting instruction is removed; and (D) for open-weight models, the stance response in the debater's own token log-probabilities. We evaluate three-model committees debating open-ended GlobalOpinionQA across 750 debates under three tones: friendly (seek common ground), neutral, and hostile (stress-test every position). (A) Tone strongly reshapes reported agreement: full agreement differs by 50.4 percentage points between the friendly and hostile endpoints. (B) A judge that reads only the reply text, never the self-report or the condition, recovers the same pattern. (C) The dissent appears partly tied to the instruction that elicited it: labels revert toward agreement 23.1 points more often after deleting the hostile instruction than under a matched re-ask that keeps it; question-weighted inference is inconclusive on first-round turns alone (p=0.0625), significant pooling all rounds (p=0.016), and only 11/28 first-round reversions also appear in the reply text. (D) Opposing arguments weaken a debater's stance margin more consistently than they shift its direction. For final answers we detect no quality gain: a bias-checked jury returns 299/299 ties (ruling out only large differences), accuracy on a verifiable control task is unchanged, and a jury without the bias check had declared debate the winner 66% of the time -- an artifact of reading order. Taken together, LLM debate readily changes what agents say, but we find much weaker evidence that it changes what they persistently endorse or improves the quality of the final answer.