When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
2026-08-10 • Computation and Language
Computation and LanguageArtificial IntelligenceHuman-Computer InteractionMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) behave when answering medical questions with uncertain or missing information. They tested models using tricky questions where either the wording was unclear or the correct answer was removed, forcing the model to admit it didn’t know. They found that while the models made more mistakes in these uncertain cases, their confidence didn’t drop accordingly, meaning the models often acted sure even when wrong. This mismatch poses risks for using such models in medical decisions. The study shows the need for better ways to evaluate and handle uncertainty before deploying these models in healthcare.
Large Language ModelsMedical Question AnsweringClinical ReasoningUncertaintyModel CalibrationExpected Calibration ErrorClinical Decision-MakingHallucination in AIEpistemic ReliabilityMedMCQA Dataset
Authors
Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam, Quan Z. Sheng
Abstract
Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks. However, their reliability under uncertainty remains poorly understood which raises critical concerns for deployment in high-stakes clinical settings. In such environments, incorrect predictions are inherently risky, but confident incorrect predictions can be particularly harmful as they may mislead clinical decision-making. In this paper, we conduct a systematic behavioral analysis of LLMs under clinical information uncertainty. We propose an evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings. First, we introduce linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts. Second, we construct an answer removal setting, wherein the correct option is deliberately excluded mandating the model to recognize insufficient information and abstain. We analyze both model accuracy and confidence behavior using multiple calibration metrics including calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER) across 500 medical questions. Our results reveal a consistent failure mode, i.e., although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy. This leads to a substantial increase in unsafe confident errors, indicating that model confidence remains largely insensitive to clinically meaningful information loss. Furthermore, we observe significant variation across models in their ability to abstain when the correct answer is unavailable, with some models persistently producing high confidence hallucinated answers. These findings expose critical limitations in the epistemic reliability of current LLMs and highlight the need for uncertainty aware evaluation methods prior to their deployment in clinical workflows.