Correctness criteria shape clinical uncertainty detection in vision language models
Rethinking Correctness for Uncertainty Estimation in Clinical Prediction with Vision-Language Models
Machine Learning
Summary
Models that combine images and text are used to help predict health outcomes from medical records, but they sometimes make mistakes. It’s important to know when their predictions are unreliable. The authors show that how we decide if a model’s prediction is right or wrong can change how well we find errors. They tested several ways to check correctness and found some match human judgment better and avoid misleading uncertainty estimates. This means checking how we measure correctness is important before trusting these models in healthcare.
What this means in practice
- •For hospital data teams: Choose and validate correctness criteria to improve detecting unreliable clinical predictions from combined image-text models.
- •For medical ai developers: Select best evaluation methods for uncertainty estimation to avoid misleading assessments when deploying clinical vision-language models.
Authors
Mingcheng Zhu, Jinning Liang, Tingting Zhu
Abstract
Vision-language models are increasingly explored for clinical prediction from electronic health records and medical images, where identifying unreliable predictions is important for safe deployment. Uncertainty estimation (UE) enables detecting such predictions, but its evaluation depends on a correctness criterion that determines whether each model output is correct. If this criterion disagrees with human judgement or distorts downstream UE performance, conclusions about model reliability can be misleading. We introduce a two-axis framework that evaluates correctness criteria by their agreement with human judgements and fidelity to human-referenced UE performance. We assess eight criteria across three clinical prediction tasks and three models using 450 predictions annotated by two reviewers. Across the audited tasks, canonical exact matching (EM) achieved the highest observed human agreement and lowest UE distortion, while the BERT-based matching (BEM) and LLM-judge also showed strong human agreement. Across four UE methods and 23,254 clinical predictions, criterion choice changed error-detection AUROC by up to 0.146 and reversed the relative ranking of UE methods. The LLM-judge also selectively accepted invalid or uncertain outputs, accepting 16 of 30 such human-identified errors. These results demonstrate that correctness assessment is an integral component of clinical UE evaluation and should be validated before UE methods are compared.