Self-attributing vision transformer improves explainability in medical imaging
SAP-ViT: A Self-Attributing Patch Vision Transformer for Medical Image Classification
Artificial Intelligence
Summary
Medical image classifiers usually decide what the image shows using one overall summary token, which makes it hard to understand which parts of the image mattered. The authors propose SAP-ViT, a model that predicts the class by adding up contributions from many small image patches directly. This allows the model to explain its decisions by showing which local areas influenced the result. The approach keeps strong accuracy while giving clearer and more precise explanations of the medical images.
What this means in practice
- •For medical imaging teams: Improve medical image classifiers by integrating patch-level explanations that align with clinical interpretability standards.
- •For healthcare ai developers: Develop diagnostic AI tools that provide accurate predictions alongside clear, patch-based visual evidence to aid medical decision making.$Commercial implications: Enables sale of AI diagnostic software that enhances trust by showing interpretable image region contributions to clinicians.
Authors
Wei Zhang, Wolfgang Nejdl, Ming Tang
Abstract
Vision Transformer (ViT) classifiers commonly rely on a global classification (CLS) token for image-level prediction, which is poorly aligned with local interpretability requirements in medical imaging. We propose SAP-ViT, a Self-Attributing Patch Vision Transformer that replaces CLS-based classification with direct aggregation of last-layer patch embeddings. This design makes class predictions decomposable into patch-level contributions as part of the forward pass. To encourage concise and readable evidence, SAP-ViT applies sparsity regularization to the patch-weight distribution. Since self-attention mixes information across patch tokens, we further introduce an input-space back-mapping mechanism to estimate which input patches contribute to influential output patch embeddings. We evaluate SAP-ViT qualitatively and quantitatively on multiple medical image classification datasets spanning different imaging modalities. The results show that SAP-ViT maintains strong classification performance while providing consistent and interpretable patch-level explanations across datasets. Ablation results further show that both sparsity regularization and the alignment component of input-space back-mapping contribute to the final explanation quality. Overall, SAP-ViT provides a favorable trade-off between predictive accuracy and interpretability for ViT-based medical image classification.