Localization lens improves medical vision language model accuracy by six percent
Localization Lens for Improving Medical Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical AI systems that analyze images and text often struggle with understanding exactly where body parts are and how they relate spatially. The authors designed a new method called localization lens that adds expert-verified anatomical details to help these models better grasp anatomy and positioning. They also improved the AI’s internal processing and training so that these details are used more effectively. Tests showed their method helped medical question-answering systems better understand anatomy and location, improving accuracy by up to 6.2%. Their approach can be added to many existing medical vision-language models.
What this means in practice
- •For medical imaging software developers: Improve medical image question-answering tools by integrating localization lenses to enhance anatomical and spatial understanding.
- •For clinical decision support teams: Enhance clinical AI systems’ accuracy in interpreting medical images by incorporating better spatial and anatomical reasoning.
Authors
Hasan Farooq, Murtaza Taj, Mehwish Nasim, Arif Mahmood
Abstract
Medical Vision-Language Models (Med-VLMs) have demonstrated strong capabilities in clinical tasks. However, they often struggle to understand anatomical structures and spatial positioning, which are crucial for medical reasoning. To address this, we propose a localization-aware enhancement to the Med-VLM pipeline, introducing improvements at three levels: data,architecture, and alignment. First, we introduce localization lens, a set of expert-validated representations that provide richer anatomical and positional context. However, as these representations increase input complexity, we integrate pixel shuffle within the model architecture to filter and refine representations, enhancing spatial information processing while preserving anatomical continuity. Lastly, to effectively align the localization lens representations with textual features, we incorporate decoupled contrastive loss (DCL) alongside the standard loss function. This ensures better feature discrimination and robustness, particularly in data limited medical settings. Through extensive evaluations on medical visual question answering (Med-VQA) datasets, we show that our methodology improves localization-driven performance across different Med-VLM architectures. Our analysis of localization-based questions further reveals that improvements in anatomy and spatial reasoning directly enhance the overall accuracy of Med-VQA upto 6.2%. The proposed approach is model-agnostic and can be seamlessly integrated into existing Med-VLM pipelines. The dataset, code, and trained models will be made publicly available at https://github.com/CVLABLUMS/localizationlens.