Geometry guided learning improves medical image lesion interpretation
Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs
Computer Vision and Pattern Recognition
Summary
Medical AI models have trouble understanding images with lesions that show multiple disease traits at once. The authors created a new method called PureVision that teaches these models to better recognize the shape and location relationships of different lesion features. It uses special geometry-based guidance to preserve important anatomical details while learning. This helps the AI provide more accurate answers and reports about complex lesions in medical images.
What this means in practice
- •For medical imaging ai developers: Enhance clinical image interpretation tools to better analyze lesions with multiple pathological traits using geometry-informed visual embeddings.
- •For radiology software engineers: Improve radiology report generation and visual question answering systems by integrating geometry-supervised visual representations of multi-phenotype lesions.
Authors
Hao Wang, Qiwei Zeng, Shuchang Ye, Jinghao Lin, Yuezhe Yang, Yige Peng, Jinman Kim, Lei Bi
Abstract
Medical vision-language models (VLMs) have shown increasing potential for clinical image interpretation. However, these models still struggle to interpret multi-phenotype lesions whose diagnosis requires the joint assessment of multiple pathological phenotypes. Existing vision-language alignment methods produce visual representations that fail to preserve anatomical hierarchies and relationships among phenotypic subclasses. This stems from their reliance on semantic supervision, which lacks geometric constraints to preserve these relationships in the visual embedding space. Moreover, the sparsity of lesion-related anatomical and phenotypic representations makes it difficult for medical VLMs to capture important diagnostic evidence. To address these limitations, we propose \textbf{PureVision}, a geometry-supervised visual representation learning framework for multi-phenotype lesion interpretation in medical VLMs. It combines a geometry-supervised representation learning module, \textbf{PureEyes}, and an anatomy-guided evidence aggregation module, \textbf{PureNeurons}. PureEyes provides geometric supervision through ideal spatial distributions that encode anatomical hierarchies and phenotypic subclass relationships. PureNeurons projects visual representations into the learned latent space, using their positions to selectively aggregate lesion-specific anatomical and phenotypic evidence. Experiments on \textit{LIDC-IDRI}, \textit{CBIS-DDSM}, and \textit{3DReasonKnee} demonstrate that PureVision improves lesion grounding and phenotype characterization in visual question answering and radiology report generation. Code is available at: https://anonymous.4open.science/r/purevision-06C2.