Summary
Detecting unusual or unexpected medical images is important to keep AI systems safe and reliable. The authors found that using features from only the deepest layer of vision-language models may miss important signals found in the middle layers, which vary depending on the type of medical image. They also showed that previous methods to pick the best layers were unstable and could greatly affect performance. To fix this, the authors introduced a method that looks at multiple ways to measure uncertainty, making the layer selection more stable and improving detection. Their approach worked better across different medical datasets and AI models without needing extra training.
out-of-distribution detectionvision-language modelsmedical imaginglayer selectionentropy estimationzero-shot learningcross-modal similarityAUROCdomain shiftlatent space
Authors
Shyam Nandan Rai, Francesco Di Salvo, Sebastian Doerrich, Christian Ledig
Abstract
Out-of-distribution (OOD) detection is crucial for safe deployment of medical AI systems, where domain shifts arise across institutions, acquisition protocols, and patient populations. VLMs enable zero-shot OOD detection by embedding images into a language-aligned latent space, where cross-modal similarity serves as a non-parametric confidence signal for identifying in-distribution samples. Yet existing methods rely almost exclusively on final-layer embeddings, implicitly assuming that the deepest representations are universally optimal. We first show that this assumption does not hold in medical imaging: intermediate layers provide complementary OOD signals, and the optimal representational depth depends on the respective image modality. While prior work selects layer combinations via entropy minimization of normalized histograms, we demonstrate that single-resolution entropy estimation is highly sensitive to binning choices, leading to performance variations of up to 19.3% AUROC. To address this instability, we propose a multi-resolution entropy estimation strategy that aggregates histogram statistics across multiple discretization scales, enabling robust and stable intermediate-layer selection. Across two medical OOD benchmarks, namely MIDOG and OASIS, covering distinct imaging modalities, diverse shift types, and different VLM backbones, our method consistently outperforms state-of-the-art approaches, offering a lightweight and stable solution for zero-shot OOD detection.