MorphoSHAP explains deep visual models using shapes not pixels
MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Most tools that explain AI vision models highlight important pixels or patches, but this often misses how those parts relate as structures. The authors created MorphoSHAP, which breaks images into meaningful shapes and measures how each shape contributes to a model’s decision. This approach shows not only where the model looks but also what types of shapes influence its prediction and by how much. MorphoSHAP works across different models and datasets and provides explanations that people found easier to understand and prefer.
What this means in practice
- •For computer vision engineers: Use shape-based explanations to better understand and debug deep visual models beyond pixel-level heatmaps.
- •For ai system auditors: Provide clearer, structured explanations of AI image predictions for compliance and trust assessments.
Authors
Anirudh Prabhakaran, Alexandre Rocchi, Gianni Franchi
Abstract
Visual attribution methods typically explain predictions using pixels, superpixels, or regular patches. These representations can localize important regions, but provide limited information about their structure. We introduce MorphoSHAP, a model-agnostic post-hoc method that instead uses morphological shapes as the players of a Shapley attribution game. Using the Tree of Shapes, each shape is described by its scale, geometry, and signed contribution, providing explanations of where the evidence lies, what type of structure carries it, and how strongly it affects the prediction. This shared morphological vocabulary enables spatial, textual, and global class-level explanations beyond image-specific heatmaps. To the best of our knowledge, MorphoSHAP is the first SHAP-based image attribution framework to combine these different forms of explanation. Across five diverse datasets and three architectures, MorphoSHAP achieves strong insertion/deletion performance and outperforms competing attribution methods on several benchmarks. Finally, a user study shows that MorphoSHAP provides explanations that are easy to use and are preferred over standard attribution baselines.