Papers for

healthcare safety teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Clinical language models show bias and inconsistent answers beyond accuracy

Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMs

Abstract: Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustness. We evaluated six LLMs on 150 MedQA USMLE questions using two automated perturbation tests to assess their performance. The counterfactual validity (CFV) test asked each model to make a minimal, plausible clinical change that would make a different answer correct. The demographic robustness test added six demographic prefixes to the same vignette and compared the answers and explanations with a no demographic baseline. Of the 900 CFV attempts, 228 (25.3 %) were valid and 672 were invalid. Across 5,400 demographic comparisons, 1,097 answers were changed (20.3%). Automated judging identified 3,128 stereotype evidence flags, including 1,932 in the broad Other category. MedGemma 27B achieved the highest accuracy (87.1%) and CFV (63.3%), lowest answer change rate (16.0%), and low mean Explanation Demographic Dissonance (EDD) score (0.169). However, its accuracy still exceeded its CFV, indicating that correct answers do not guarantee reliable performance on the counterfactual validity task. OpenBioLLM had the highest answer change rate and EDD, whereas GLM had the highest stereotype flag rate. These findings show that accuracy, CFV, answer stability, EDD, and stereotype evidence capture different evaluation aspects. Because all judgments were automated and no clinician validation was available, the results support safety screening but do not establish clinical deployability of the model.

Sat 26 SeptArtificial Intelligence
The gist
Clinical language models are often tested only for accuracy, but this paper shows that accuracy alone doesn't catch problems with changing clinical details or patient demographics. The authors tested six language models on medical exam questions by changing patient information or details to see if answers would change appropriately and if bias appeared. They found many cases where answers were unstable or changed unfairly with demographics, revealing hidden weaknesses. The work highlights that accurate answers don’t guarantee reliable or unbiased clinical performance in these models.
Open → 2609.32807v1