Medical diagnosis accuracy drops using multi-turn patient dialogues
MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Artificial Intelligence
Summary
Most medical AI tests use single questions and multiple-choice answers, which are very different from real doctor visits where many questions are asked over time. The authors created a test using realistic doctor-patient conversations based on real exam cases, with patients acting in different ways. They found that AI doctors do much worse when they have to do multiple back-and-forth questions, and that differences in patient background affect the AI's accuracy. This shows current tests miss important challenges seen in real medical consultations.
What this means in practice
- •For clinical ai developers: Evaluate AI diagnostic systems under realistic multi-turn doctor-patient conversations to better assess performance in real-world settings.
- •For medical chatbot designers: Improve chatbot behaviour by incorporating patient persona effects and question relevance over multiple turns during simulated medical consultations.
Authors
Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang, Rafiq Ali, Bdour Babillie, Nazira Dunbayeva, Lang Gao, Omar Hussein, Ahmed Nada, Ahmed Mohamed Magdy Mohamed, Jinghui Liu, Salman Khan, Imran Razzak, Yuxia Wang, Xiuying Chen
Abstract
Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss.