Medical fine-tuning changes but does not lower AI risk for patient answers
Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework
Artificial IntelligenceComputation and Language
Summary
Medical question-answering AI can give answers that are factually correct but still risky if followed by patients. The authors created a way to measure different kinds of risks in AI answers, like dangerous treatment advice or overconfident statements. They found that fine-tuning AI models on medical data changes what kinds of risks appear, but doesn’t make the overall risk lower. The study also shows which risk types are easier or harder to judge automatically.
What this means in practice
- •For healthcare ai developers: Use the rubric and judge prompts to evaluate patient-facing medical AI for diverse safety risks beyond factual correctness.
- •For clinical risk assessment teams: Assess how medical fine-tuning changes the types of risks in AI-generated medical answers to better target risk mitigation strategies.
Authors
Savan Doshi
Abstract
Patient-facing medical question answering with large language models (LLMs) is moving from research demonstrations to production deployments, yet evaluation remains anchored on factual correctness. Factuality is necessary but not sufficient: a response can be factually grounded and still issue a directive that, if acted on, would harm the patient. We introduce a risk-sensitive evaluation framework that scores responses by the presence and severity of six categories of risk-bearing language (treatment directives, contraindications, dosage expressions, urgency-triage cues, high-alert medication mentions, and overconfident assertions), anchored on published clinical harm taxonomies. Severity is classified by a three-judge LLM ensemble. We apply this framework to six 7-9B instruction-tuned models, three general (Mistral-7B, Phi-3.5-mini, Llama-3.1-8B) and three medically adapted (BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B), on 200 real patient questions from the K-QA benchmark. Two findings emerge. First, ensemble risk-bearing language is statistically independent of factual hallucination rate (pooled Spearman $ρ= -0.065$, $n = 300$, 95% CI $[-0.18, +0.05]$; distance correlation $= 0.10$, independence test $p = 0.24$). Second, medical fine-tuning does not reduce aggregate risk but reshapes it: continued-pretraining adaptations (BioMistral, OpenBioLLM) consistently suppress high-alert medication mentions while amplifying overconfident assertions, whereas clinical instruction-tuning (Med42) suppresses dosing only, without amplifying overconfidence. Per-category Krippendorff $α$ is fair to good for the four linguistically anchored categories (0.53-0.71) and low for the two clinical-judgment-dependent categories (0.19, 0.31), indicating where automatic safety evaluation is and is not yet reliable. We release the rubric, judge prompts, and evaluation code.