AI models provide safer second opinions for complex medical cases
AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions
Computation and LanguageArtificial Intelligence
Summary
When doctors face difficult cases, getting a second opinion helps ensure good care. The authors created a test to see how well AI language models can give second opinions that are accurate, clear, and safe to follow. They found that combining different AI roles into a team-like 'consilium' improved the advice quality, especially when using structured reports. The best model was faster and more consistent in agreeing on diagnoses and treatments.
What this means in practice
- •For clinical ai developers: Build AI systems that offer reliable second opinions by integrating multiple role-specific expert views to improve clinical decision support quality and safety.
- •For healthcare it teams: Implement structured report formats in AI clinical systems to enhance clarity and management recommendations for complex patient cases.
Authors
Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev, Mikhail Yu. Valkov, Valeria Yu. Chernina, Natalia Gombolevskaya, Ayyaz Hussain, Vladimir M. Agafonov, Victor A. Gombolevskiy, Aleksandr V. Nesterov, Iaroslav R. Bespalov, Ivan V. Oseledets, Dmitry V. Dylov
Abstract
Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, and safe. We developed a multidimensional benchmark using 300 diagnostically resolved NEJM--MedQA cases: 110 for physician reference ratings and Judge calibration and 190 for held-out evaluation. Eight single-model pipelines generated independent role-specific opinions and interactive-consilium reports under free-form and structured clinical-reasoning conditions. A physician-calibrated LLM Judge evaluated 3,040 final reports and the corresponding structured primary-role outputs. Model identity separated report quality more strongly than response format. Structured final reports had higher overall mean scores in seven of eight model pipelines, with their clearest advantage in management (2.740 vs. 2.436). Compared with the structured pre-consilium primary-role opinion, the complete structured synthesis had higher pooled scores across all five shared criteria, with the largest increases in problem prioritization, factual grounding, and reasoning coherence. GPT-5.6-Sol led every evaluated component and reached consensus in the fewest review rounds. These findings show that the benchmark can distinguish model capability, the contribution of multidisciplinary synthesis, criterion-specific effects of response structure, deliberation dynamics, and inference requirements within one common-case evaluation.