Interpretable transformer improves accuracy and adaptability across domains
INTR++: INterpretable TRansformer with better Accuracy, Interpretability, and Adaptability
Computer Vision and Pattern Recognition
Summary
Interpretable Transformers are models that show how they make decisions but usually work well only in specific fields. The authors found that existing methods with fixed prompts struggle when images vary a lot across different areas like medical scans or satellite photos. They created INTR++, which adds a small module to make prompts adapt to different types of images, keeping explanations clear and accurate everywhere. This helps the model maintain good performance and interpretability across many kinds of data without much extra complexity.
What this means in practice
- •For medical imaging teams: Improve interpretability and consistent performance of AI models across various medical image types using adaptive prompts.
- •For remote sensing analysts: Provide reliable and interpretable AI results for diverse satellite and aerial imagery with adaptive prompt adjustment.
Authors
Dipanjyoti Paul, Arpita Chowdhury, Maneet Chatterjee, Daichi Kitaguchi, Tanya Berger-Wolf, Tetsuya Sakurai, Wei-Lun Chao
Abstract
Interpretable Transformers often perform well in domain specific settings but lose effectiveness beyond that. Achieving ViT-like broad applicability requires models with consistent performance across domains. Recent works on Interpretable Transformers, INTR and Prompt-CAM, were developed keeping domain adaptability in mind. Prompt-CAM leverages learnable prompts to promote adaptability. However, these prompts are fixed in nature and therefore struggle to adapt to domains with substantial variation in input image characteristics, resulting in reduced interpretability. To address this limitation, we propose INTR++, which extends Prompt-CAM by incorporating a lightweight Prompt Adapter Module (PAM). PAM enables prompts to be adaptive by enriching them with domain-specific knowledge, thereby ensuring consistent interpretability across diverse domains including those with varying image characteristics such as medical imaging, micro-scopic imaging, and remote sensing imagery. INTR++ retains the standard architecture and end-to-end learning of Interpretable Transformer based models, while adding only a small number of additional parameters compared to Prompt-CAM. We demonstrate that INTR++ delivers consistent improvements in all aspects across domains like animals, plants, medical imaging, microscopic imaging, remote sensing imagery and object-centric data.