Clinical language models show bias and inconsistent answers beyond accuracy
Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMs
Artificial Intelligence
Summary
Clinical language models are often tested only for accuracy, but this paper shows that accuracy alone doesn't catch problems with changing clinical details or patient demographics. The authors tested six language models on medical exam questions by changing patient information or details to see if answers would change appropriately and if bias appeared. They found many cases where answers were unstable or changed unfairly with demographics, revealing hidden weaknesses. The work highlights that accurate answers don’t guarantee reliable or unbiased clinical performance in these models.
What this means in practice
- •For clinical ai developers: Use automated counterfactual and demographic tests to identify bias and instability in clinical language model outputs before deployment.
- •For healthcare safety teams: Screen language models with demographic and counterfactual robustness tests to support patient safety efforts in clinical AI tool assessment.
Authors
Chaitai Deb Purkayastha, Bharath Kumar Bolla, Vishnu Surya Reddy Nandi
Abstract
Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustness. We evaluated six LLMs on 150 MedQA USMLE questions using two automated perturbation tests to assess their performance. The counterfactual validity (CFV) test asked each model to make a minimal, plausible clinical change that would make a different answer correct. The demographic robustness test added six demographic prefixes to the same vignette and compared the answers and explanations with a no demographic baseline. Of the 900 CFV attempts, 228 (25.3 %) were valid and 672 were invalid. Across 5,400 demographic comparisons, 1,097 answers were changed (20.3%). Automated judging identified 3,128 stereotype evidence flags, including 1,932 in the broad Other category. MedGemma 27B achieved the highest accuracy (87.1%) and CFV (63.3%), lowest answer change rate (16.0%), and low mean Explanation Demographic Dissonance (EDD) score (0.169). However, its accuracy still exceeded its CFV, indicating that correct answers do not guarantee reliable performance on the counterfactual validity task. OpenBioLLM had the highest answer change rate and EDD, whereas GLM had the highest stereotype flag rate. These findings show that accuracy, CFV, answer stability, EDD, and stereotype evidence capture different evaluation aspects. Because all judgments were automated and no clinician validation was available, the results support safety screening but do not establish clinical deployability of the model.