Medical question answering shows tradeoffs between models and language ai
Benchmarking decision models against generative large language models on medical multiple-choice questions
Computation and LanguageArtificial Intelligence
Summary
Medical professionals often have to pick the best answer from several choices. The authors compared traditional decision models with new large language AI models on a medical quiz. AI models were generally more accurate but cost more and took more time. Traditional models were less accurate but had advantages in cost and consistency. Choosing the right model depends on balancing accuracy, cost, speed, and reliability.
What this means in practice
- •For clinical software developers: Integrate AI configurations to improve accuracy in automated medical multiple-choice question answering workflows with cost and speed considerations.
- •For healthcare it managers: Select decision-making tools that best balance accuracy, cost, and response time to optimize clinical support systems using medical question data.
Authors
Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather
Abstract
Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.