Medical image models learn clearer signs of diseases using geometry
$Δ$Representation: Geometry Supervised Representation Learning of Phenotypes via Counterfactual Reasoning for Medical VLMs
Computer Vision and Pattern Recognition
Summary
Medical images show both normal body parts and abnormal changes like lesions, which can help doctors diagnose diseases. Previous methods mix these normal and abnormal parts when trying to teach computers to understand images. The authors created a new way that teaches computers to see the difference between normal anatomy and disease-related changes by using geometric clues and comparing what the image looks like with and without the lesion. This approach helps the computer better recognize and describe disease features in medical images.
What this means in practice
- •For medical imaging developers: Build diagnostic AI that better distinguishes disease features from normal anatomy in medical images for more accurate interpretation.
- •For radiology software engineers: Improve lesion localization and description features in radiology tools by modeling changes from normal anatomy to disease states.
Authors
Hao Wang, Qiwei Zeng, Jinghao Lin, Shuchang Ye, Yuezhe Yang, Yige Peng, Haoyuan Che, Jinman Kim, Lei Bi
Abstract
Medical vision-language models (VLMs) have shown increasing potential for radiological image interpretation. Medical VLMs encode radiological images into visual representations that capture both anatomical and phenotypic information for diagnosis. Existing approaches improve pathological phenotype representations through semantic-guided representation alignment. However, pathological phenotypes arise as lesion-specific visual changes superimposed on underlying normal anatomy. Such semantic alignment approaches fail to model the phenotype-specific increment relative to the corresponding normal anatomical representation. To address this gap, we propose \textbf{$Δ$Representation}, a visual phenotype representation learning framework based on counterfactual reasoning for medical VLMs. It comprises \textbf{BaseAnatomy}, a geometry-supervised representation learning module, and \textbf{$Δ$Phenotype}, a counterfactual incremental representation learning module. BaseAnatomy provides fine-grained geometric supervision through spatial relationships across and within anatomical structures. $Δ$Phenotype computes the representation increment between lesion representations and their corresponding normal anatomical representations, and supervises increments associated with the same phenotype to cluster in the representation space. Experiments on \textit{ReXGroundingCT} and \textit{LIDC-IDRI} demonstrate that $Δ$Representation effectively structures pathological phenotype representations and improves lesion grounding and phenotype characterization accuracy in medical VLMs. Code is available at https://anonymous.4open.science/r/deltarep-CF6D.