Combining models improves medical image anomaly detection and localization
Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Detecting small medical problems in images can be hard because existing AI models either focus on understanding overall meaning or on capturing detailed parts of the image. The authors combined two types of AI models—one that understands image and text pairs well, and one that learns detailed anatomical patterns—to better find and outline lesions. They also developed techniques to get higher resolution details and remember patterns from a few examples. Their new method performed better at spotting and outlining medical anomalies in scans of the liver, eye, and brain.
What this means in practice
- •For medical imaging software developers: Create improved tools that detect and precisely outline abnormalities in medical scans with fewer labeled examples.
- •For clinical ai integration teams: Integrate spatially aware anomaly detection components to enhance diagnosis support systems handling diverse imaging modalities.
Authors
Juzheng Miao, Yuchen Yuan, Cheng Chen, Pheng-Ann Heng
Abstract
Vision-Language Models such as CLIP enable effective few-shot medical anomaly detection (AD) via strong image-text semantic alignment. However, their globally contrastive pretraining lacks explicit spatial supervision, limiting precise lesion localization. In contrast, Vision Foundation Models (VFMs) such as DINO learn spatially coherent patch representations via self-distillation and local-to-global consistency, better capturing fine-grained anatomical structures. Leveraging this complementarity, we propose Spatial-FAD, a spatial-aware few-shot medical AD framework that improves lesion localization by combining VFM spatial priors with CLIP semantics. Specifically, we introduce a VFM-enhanced adapter that injects a structural affinity prior derived from DINO into CLIP features. This structure-guided refinement encourages visual embeddings to better adhere to lesion boundaries while maintaining semantic alignment. To address the loss of spatial detail from patchification and the limited input resolution of CLIP, we adopt a sliding-window aggregation strategy. This generates high-resolution, spatially dense embeddings to further enhance localization granularity. Moreover, we introduce a prototype-enhanced support memory scheme to efficiently exploit the few-shot support set. This module stores compact prototypes for normal and abnormal patterns, reducing memory costs while boosting performance by fusing patch-to-prototype and image-text similarities. Extensive experiments on three benchmark datasets, including Liver CT, Retinal OCT, and Brain MRI, demonstrate that Spatial-FAD significantly outperforms state-of-the-art methods, especially in lesion segmentation. Notably, in the 4-shot scenario, our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC. Code is available at: https://github.com/JuzhengMiao/Spatial-FAD.