Uncertainty mapping improves understanding of ambiguous medical image scores
Beyond Performance Metrics: Uncertainty Mapping of Label Ambiguity in Fazekas Score Prediction
Computer Vision and Pattern Recognition
Summary
Medical images are often labeled by experts, but sometimes these labels can be uncertain or inconsistent, especially when different experts disagree. The authors developed a new way to look at where a machine learning model is unsure about predicting a Fazekas score, a rating for brain damage seen on scans. Instead of just giving a number for confidence, their method shows where the model finds it hard to tell classes apart and where mistakes are likely. This helps spot when labels might be wrong and guides better review of challenging cases.
What this means in practice
- •For medical imaging teams: Identify ambiguous cases in brain scan datasets to prioritize expert review and improve label quality using uncertainty maps.
- •For clinical ai developers: Design better machine learning models for ordinal score prediction by selecting loss functions that yield clearer uncertainty profiles and class separation.
Authors
Susanne Schmid, Johanna Ospel, Richard Frayne, Roberto Souza
Abstract
Reference labels used to train medical image classification models are not always as certain as they may appear, and this uncertainty has implications on performance metrics. In this study, we propose a framework to analyze model performance for periventricular Fazekas score prediction that goes beyond conventional metrics. The Fazekas score is an ordinal visual rating scale used to assess the severity of white matter hyperintensities and is known to be affected by inter-rater variability. While the best Fazekas score prediction model achieved a Matthews correlation coefficient (MCC) of 0.70, performance varied across data splits and loss functions, making interpretation of model capabilities difficult. Rather than interpreting epistemic uncertainty of a model's prediction as an isolated scalar value, our approach of uncertainty mapping relates uncertainty to its position within the learned feature representation. This highlights regions of class-boundary transitions where cases appear more ambiguous and misclassifications are more likely. It also identifies potential label disagreement, including low-uncertainty misclassified cases that expert review found to be inconsistent with the original reference Fazekas score. Therefore, uncertainty mapping allows model behaviour to be examined in relation to class separation and potential model-label disagreement. Loss function choice also influenced the uncertainty profile, with some models showing clearer class separation and more localized uncertainty in ambiguous regions than others. These findings suggest that uncertainty mapping for Fazekas score predictions can support model interpretation and targeted dataset review when reference labels are affected by ambiguity/ inter-rater variability.