Explainable Uncertainty Estimation for Reliable Medical AI
Machine LearningArtificial Intelligence
Summary
The authors address the problem that doctors often don’t trust AI predictions because they don’t know when the predictions are uncertain or why. They created a method called egRUE that combines uncertainty estimation and explanation, showing not only how unsure the AI is but also which specific factors cause that uncertainty. Their tests show egRUE is more reliable and easier to understand than previous methods. Medical experts in their study trusted the AI’s correct predictions more and were less confident in wrong ones when given egRUE’s explanations. This helps make AI advice safer and clearer for healthcare decisions.
Authors
Li Rong Wang, Jamie Duell, Xinran Xu, Thomas C. Henderson, Yu Yue Hew, Pik Wan Erica Chiang, Xiao Wei Alstar Ang, Bingwen Eugene Fan, Xiuyi Fan
Abstract
Artificial intelligence has strong potential to support clinical decision-making, yet its adoption in healthcare remains limited due to a lack of trust. Uncertainty estimation can signal unreliable predictions, and explainable AI (XAI) can clarify how predictions are made but existing methods treat them separately, providing no feature-level insight into why a prediction is uncertain or which tests to prioritize to reduce it. To address this gap, we propose explainable uncertainty estimation, which unifies uncertainty estimation and XAI to both quantify uncertainty and explain feature-level contributions. We introduce the Expected Gradients Reconstruction Uncertainty Estimate (egRUE), which incorporates prediction explanations into its uncertainty computation and decomposes uncertainty into feature-wise contributions. We prove theoretical properties of egRUE and show through experiments that it improves reliability and interpretability compared to existing methods. A user study with medical experts further demonstrates that egRUE's explanations improve calibrated trust over uncertainty scores alone, increasing confidence in correct predictions and reducing confidence in incorrect ones. By combining prediction uncertainty with feature-level explanations, egRUE strengthens decision-making support in safety-critical healthcare settings, clarifying both when predictions may be unreliable and which features drive that uncertainty.