Clinical reasoning evaluation rubrics for large language models mapped
A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
Computation and LanguageArtificial Intelligence
Summary
The study looks at how we check if AI language models can think through medical cases properly, not just get exam questions right. The authors found that current tests don’t cover all the important parts of medical reasoning involving patient histories and decision making over time. They suggest combining existing tools and developing new tests for the trickier skills, like dealing with uncertainty and checking if reasoning is trustworthy. Their work guides better ways to assess AI’s clinical reasoning but does not provide a complete test itself.
What this means in practice
- •For clinical ai developers: Combine existing clinical reasoning tests to better evaluate AI models handling patient histories over time.
- •For medical benchmark designers: Design new evaluation tools focusing on uncertainty, counterfactuals, and reasoning faithfulness for clinical text generation.
A survey. It maps existing work.
Authors
Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo
Abstract
Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.