Vision language models improve unseen medical image detection accuracy
Layer Selection in VLMs for Zero-Shot OOD Detection via Multi-Resolution Entropy Estimation
Computer Vision and Pattern Recognition
Summary
Detecting unusual or unexpected medical images is important for ensuring AI tools work safely in hospitals. The researchers found that using information from different layers inside vision-language AI models, instead of just the final layer, helps detect these unusual images better. They also created a new way to measure the uncertainty in different layers that is more stable and reliable. Testing on medical imaging benchmarks, their method consistently improved detection of out-of-distribution scans without needing extra training.
What this means in practice
- •For medical imaging developers: Improve zero-shot detection of unexpected medical images across various scanners and patient populations using intermediate layers of vision-language models.
- •For security system engineers: Enhance anomaly detection by applying multi-resolution entropy estimation to intermediate layers of multimodal AI systems for robust confidence scoring.
Authors
Shyam Nandan Rai, Francesco Di Salvo, Sebastian Doerrich, Christian Ledig
Abstract
Out-of-distribution (OOD) detection is crucial for safe deployment of medical AI systems, where domain shifts arise across institutions, acquisition protocols, and patient populations. VLMs enable zero-shot OOD detection by embedding images into a language-aligned latent space, where cross-modal similarity serves as a non-parametric confidence signal for identifying in-distribution samples. Yet existing methods rely almost exclusively on final-layer embeddings, implicitly assuming that the deepest representations are universally optimal. We first show that this assumption does not hold in medical imaging: intermediate layers provide complementary OOD signals, and the optimal representational depth depends on the respective image modality. While prior work selects layer combinations via entropy minimization of normalized histograms, we demonstrate that single-resolution entropy estimation is highly sensitive to binning choices, leading to performance variations of up to 19.3% AUROC. To address this instability, we propose a multi-resolution entropy estimation strategy that aggregates histogram statistics across multiple discretization scales, enabling robust and stable intermediate-layer selection. Across two medical OOD benchmarks, namely MIDOG and OASIS, covering distinct imaging modalities, diverse shift types, and different VLM backbones, our method consistently outperforms state-of-the-art approaches, offering a lightweight and stable solution for zero-shot OOD detection.