Saliency methods show limits in explaining breast MRI AI decisions
Faithful Faithfulness Evaluations: Challenges & Pitfalls Learned from a Breast MRI Case Study
Computer Vision and Pattern RecognitionHuman-Computer InteractionMachine Learning
Summary
Deep learning models for medical images like breast MRIs can be hard to understand because they act like black boxes. The authors looked at ways to explain these models by highlighting important image areas, called saliency maps. They found that common ways to check if these maps really show how the model thinks can give very different results depending on how the test is done. They also showed that it matters whether explanations focus on specific disease types or not. Their work suggests that we need better, more careful ways to test if these AI explanations are trustworthy.
What this means in practice
- •For clinical ai developers: Design evaluation protocols that separate class-specific and class-agnostic explanations to better validate AI interpretability methods in medical imaging.
- •For radiology software engineers: Implement more robust saliency evaluation strategies that consider perturbation type to improve trust in breast MRI AI diagnostic tools.
Tested on one dataset.
Authors
Peachapong Poolpol, Henrik H. J. Detjen, Eike Petersen
Abstract
Saliency maps are widely used to explain deep learning predictions in medical imaging, yet visually plausible explanations do not necessarily reflect a model's true decision process and may therefore mislead clinicians. We investigate this problem using a Vision Transformer-based breast MRI classifier trained on the ODELIA Breast MRI Challenge dataset and evaluate multiple saliency methods, including Last-layer Attention, Attention Rollout, Grad-SAM, Gradient Attention Rollout, GMAR, Grad-CAM, and HiResCAM. Our study highlights two often-overlooked challenges in perturbation-based faithfulness evaluation. First, method rankings depend strongly on the perturbation strategy, varying across intensity-based perturbations and transformer-based attention masking. Second, benchmarking saliency methods requires distinguishing between class-specific and class-agnostic explanations. To enable fair comparisons, we introduce non-class-specific variants of gradient-based methods and evaluate both settings separately. Across protocols, Grad-CAM and Gradient Attention Rollout consistently emerged as the strongest class-specific methods, although their relative ranking depended on the evaluation design. These findings expose important limitations of current saliency-based explainability approaches and highlight the need for more robust and standardized evaluation frameworks for trustworthy clinical AI systems.