A Source-Grounded Framework for Constructing and Evaluating Progressive Multimodal Diagnostic Dialogues from Clinical Case Reports
2026-08-24 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created a new way to turn detailed medical case reports into step-by-step conversations that mimic how doctors diagnose patients using history, exams, images, and tests. They also made a method to test if AI models can not only give the right diagnosis but also explain their reasoning and interpret medical images correctly. When tested, their framework accurately matched the real case details, but current AI models struggled to reason and interpret as well, even if their answers sounded good. This shows that sounding confident doesn’t mean the AI understands the evidence, and the authors’ approach helps spot these gaps.
multimodal diagnosismedical benchmarksdiagnostic reasoningmedical imagesmultimodal large language modelsF1 scoreclinical case reportsevidence-grounded reasoninginteractive diagnostic agentsevaluation framework
Authors
Yufan Wang, Rui Yang, Yi Liu, Yi Lin, Yifan Peng
Abstract
Clinical diagnosis requires progressive integration of patient history, physical examination, laboratory findings, medical images, and diagnostic-informative tests. However, most multimodal medical benchmarks evaluate fixed inputs or endpoint answers, while fully interactive diagnostic agents conflate evidence selection with evidence interpretation. We present a source-grounded framework to construct progressive multimodal diagnostic dialogues from case reports and an evaluation strategy for assessing MLLMs on final diagnosis, diagnostic reasoning, and image-finding interpretation. Evaluation on 24 internal medicine case reports showed that our framework can accurately convert case reports into reference dialogues, achieving a diagnosis F1 of 0.99 and a reasoning-quality score of 4.79 out of 5. Evaluation on two frontier MLLMs (o4-mini and Claude Haiku 4.5) achieved reasoning-quality scores of 2.75 and 2.50, respectively, with substantially lower diagnosis, reasoning, and image-finding F1 scores. The results demonstrate that fluent responses do not necessarily reflect evidence-grounded clinical reasoning and highlight the utility of the proposed framework for evaluating multimodal diagnostic reasoning.