Bayesian method improves confidence and trust in multiple AI models’ decisions
Counting Moves, Weighing Voices: Bayesian Dialectical Argumentation for Calibrated Multi-LLM Councils under Persistent Adversaries
Artificial Intelligence
Summary
When several AI language models work together to answer questions, it’s important to know how likely their combined answer is correct. Existing ways to combine their answers tend to reflect how decisive the group is, not how accurate, and they struggle if some models are always unreliable. The authors created a new method that watches how these models argue—who agrees, disagrees, or changes their mind—and uses this to estimate how trustworthy each model is. This way, the combined answer comes with a better-calibrated confidence, and the system can handle models that may be deliberately misleading.
What this means in practice
- •For software engineering teams: Improve reliability and trustworthiness when combining multiple AI models’ outputs for decision-making tasks by weighting models based on their inferred reliability during deliberation.
- •For customer support automation teams: Deploy multi-model AI systems that better estimate the confidence of automatic answers while resisting persistent incorrect or biased responses from some models.
Authors
Ionel Eduard Stan, Paolo Napoletano
Abstract
A multi-LLM \emph{council} lets several large language models (LLMs) deliberate on a question and return an answer together with a confidence estimate. As these systems become increasingly used for reasoning, that confidence should represent a calibrated \emph{probability of being correct}, and the decision should remain robust when some agents are persistently unreliable. Existing \emph{council aggregation} methods fail on both fronts: their confidence estimates measure decisiveness rather than correctness, and they cannot identify or discount persistently unreliable agents. We introduce Bayesian Dialectical Argumentation (BDA), which treats the council's \emph{typed} moves---who proposed, challenged, or conceded which answer---as observations of a classical annotator model with \emph{per-agent} reliabilities. This formulation recasts multi-agent deliberation as a reliability estimation problem, using the deliberation trace to infer agent reliability under persistent adversarial behavior. By weighting evidence according to inferred agent reliability, BDA yields calibrated posterior probabilities over candidate answers while allowing persistently unreliable agents to be inverted rather than merely outvoted. Across binary and multi-class benchmarks, BDA achieves the best calibration among zero-cost council aggregation methods, requiring no additional LLM calls, and improves robustness under persistent adversarial coalitions while remaining competitive in clean settings.