Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors tested different ways of explaining AI decisions (called attribution methods) on both traditional CNN models and newer Vision Transformers (ViTs). They found that how well these explanation methods work depends a lot on the type of AI model used, and results from CNNs don’t always apply to ViTs. Some popular methods work well on CNNs but not on newer transformer types. They also showed that using only one way to judge explanation quality is not enough and stressed the importance of considering different factors. Their work provides a detailed and fair comparison to help improve explanations for various AI architectures.
Explainable AIAttribution methodsConvolutional Neural NetworksVision TransformersFaithfulnessLocalizationRobustnessAttention rolloutCAM methodsBenchmarking
Authors
Sathiyamohan Nishankar, Nethmi Pathirana, Pubudu Sanjeewani, Asanka Perera, Selvarajah Thuseethan
Abstract
Most evidence on the effectiveness of explainable artificial intelligence (XAI) attribution methods has been established on convolutional neural networks (CNNs), with limited investigation into whether these conclusions generalize to the diverse Vision Transformer (ViT) architectures that now dominate computer vision. This paper presents a controlled benchmark that evaluates attribution quality across five dimensions: faithfulness, localization, robustness, complexity, and computational cost. A standardized framework assesses 13 attribution methods from four algorithmic families on eight representative backbones spanning CNNs, isotropic ViTs, hierarchical transformers, hybrid architectures, and linear-attention transformers. The results show that attribution performance is strongly architecture-dependent and that rankings established on CNNs do not reliably transfer to transformer-based models. CAM-based methods achieve the highest scores under the conventional bounding-box localization metric on CNNs and most ViTs but perform poorly on linear-attention architectures. Pixel-level dense-mask evaluation further reveals that these gains largely reflect metric saturation rather than accurate localization. CAM-based methods also exhibit limited robustness on global-attention transformers, whereas attention rollout provides consistently stable explanations with poor localization. Furthermore, faithfulness correlation offers limited discrimination between attribution methods, highlighting the limitations of single-metric evaluation. These findings challenge prevailing conclusions on attribution performance and demonstrate the need for architecture-aware, multi-dimensional evaluation. The open-source code for the evaluation framework and benchmark results is available at https://github.com/Nishan-Charlie/VIT_XAI_Bench.