Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
2026-07-20 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied whether adding a special prompt helps AI medical models avoid giving overly confident answers when information is incomplete. They found that this prompt reduced unsafe overconfidence, but by different amounts depending on which AI judge was scoring the answers. While the safety improved, some models became less helpful in giving correct diagnoses. Human clinicians confirmed that the AI judge was very sensitive but not very precise. The authors suggest that safety improvements judged by AI should be seen as relative and directional, and considered alongside helpfulness, rather than taken as exact measurements.
large language modelsclinical language modelsoverconfidenceevidence sufficiencyAI judgment calibrationsafety in AIdiagnostic accuracybenchmark datasetshuman reviewhelpfulness
Authors
Koyar Afrasyab
Abstract
Background: LLM judges increasingly score whether clinical language models give overconfident answers under incomplete evidence, yet whether a measured "safety gain" reflects real behavior change or the judge's calibration is unresolved. Using a structured evidence-sufficiency prompt as a test case, we asked whether it reduces unsafe overconfident answers, how far that effect depends on the scoring judge, and what it costs in helpfulness. Methods: In a retrospective public-data benchmark (Real-POCQi, HealthBench, MedRBench), four models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3) answered a fully paired common panel (1,200 cells) with a standard prompt and the wrapper. The pre-specified endpoint was the paired reduction in unsafe overconfidence scored by the primary judge (GPT-5.4-nano); secondary analyses added a different-family judge (Claude Sonnet 5), a correctness judge, matched scaffold controls, and a blinded three-clinician review. Results: Unsafe overconfidence fell from 49.3% to 24.7%, a paired reduction of 24.7 points (95% CI 21.8-27.7; p<0.001), robust in direction across models and paraphrases. Magnitude was judge-dependent: Sonnet agreed on direction but nearly halved the effect (+13.1 points), with one-directional disagreement. Blinded clinicians characterized the primary judge as a high-sensitivity (1.00), low-specificity (0.55) screen, not a calibrated rate. The gain carried a model-specific helpfulness cost (correct diagnosis 80.3% to 50.3%): near-free for GPT-5.5, near-total for Gemini (-58 points). Matched scaffold controls showed genuine behavior change, not judge circularity. Conclusions: LLM-judged clinical safety effects should be reported as directional and relative, anchored to human review and evaluated jointly with helpfulness, not as calibrated absolute rates. This does not establish clinical deployment readiness.