Rubric proposal evaluates clinical reasoning in AI language model answers
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
Computation and LanguageArtificial Intelligence
Summary
Language models answering medical questions need ways to judge how well they think through problems. The authors suggest a new scoring guide that combines ideas from medical training tests and language model benchmarks. This guide helps reviewers score written model answers about medical cases, showing how well the model explains its thinking. The guide isn’t final or tested yet but aims to make scoring more clear and open to improvement.
What this means in practice
- •For clinical ai developers: Use the rubric to make model answers about clinical cases more transparent and easier to compare before final tool validation.
- •For medical content reviewers: Adopt the rubric to structure and clarify human review of AI-generated medical explanations, supporting quality checks.
A position paper. It proposes an approach and reports no results.
Authors
Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo
Abstract
Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM benchmarks (MedR-Bench, HealthBench, TIMER-Bench, DR.BENCH, PrIME-LLM and PatientSafeBench); and general LLM reasoning evaluation research, including the Factuality-Validity-Coherence-Utility taxonomy, FaithCoT-Bench and C2-Faith. We use groundedness as a clinically oriented adaptation of the taxonomy's factuality category. The rubric brings these concepts together in a multidimensional framework for scoring free-text responses to gold-standard clinical vignettes. It includes provisional behavioural anchors, applicability rules and a separate flag for case-specific safety-critical errors. General-domain frameworks inform its design but are not treated as validated clinical instruments. The rubric does not replace case-specific reference criteria or the task-specific metrics of existing benchmarks. It has not yet been tested for inter-rater reliability, construct validity or clinical utility. Its immediate purpose is to make evaluation decisions explicit and open to scrutiny before empirical testing.