Medical vision language models often miss visual details in dermatology images
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical vision-language models can help with diagnosing diseases from pictures like skin images, but they sometimes don’t use the image information as well as they could. The authors found that strong parts of these models, called vision encoders, are not fully tapped into during diagnosis. They studied why this happens with skin disease images and found ways to improve the models by changing how they pay attention to images without needing lots of labeled data. Their ideas also work with different model types and other medical image tasks.
What this means in practice
- •For clinical ai developers: Improve dermatology diagnosis models by enhancing vision attention using label-free prompting and encoder-assisted reranking without retraining the full model.
- •For medical device software teams: Integrate better vision-language model techniques to boost skin disease classification accuracy while preserving cross-domain medical question-answering abilities.
Authors
Janet Wang, Yunbei Zhang, Xiao Wang, Jihun Hamm
Abstract
Medical Vision-Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable reasoning. However, a critical performance gap exists between their strong vision encoders and the full multimodal model: in dermatology, the MedSigLIP encoder outperforms MedGemma by an average of 10.26 percentage points even when both use zero target-task labels; few-shot linear probing provides further evidence of strong visual representations. This gap motivates an investigation of how visual information is used in end-to-end diagnosis and why plausible-sounding predictions can lack grounding in image evidence. Using dermatology as our primary testbed, we systematically investigate three hypotheses for this phenomenon. We further provide a mechanistic analysis of the model's internal attention patterns, showing that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation. Task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in our evaluation. To address these challenges, we combine label-free prompting with low-label encoder-assisted reranking while keeping the VLM frozen. We validate the interventions across five VLM backbones in dermatology and provide supporting representation and attention analyses across additional medical modalities.