Concept guided model improves lung lesion images for better care
CGSM: Concept-Guided Segmentation Model for Precise Pulmonary Lesion Delineation
Computer Vision and Pattern Recognition
Summary
Precise outlines of lung problems in medical images are important for good diagnosis and treatment. The authors noticed that previous methods often miss details because they only use rough text labels that don't fully explain the shapes or small parts of the problem. They created a model called CGSM that uses detailed concept information from expert-reviewed text combined with image data to better pinpoint lesions. Their system helps the computer focus on image features that match these concepts and combines text and visuals more effectively. Tests show this method works better than others in marking lung lesions accurately.
pulmonary lesionimage segmentationconcept guided modellarge language modelsemantic guidanceimage-text fusionDice coefficientmIoUmedical imagingcomputer vision
Authors
Changheng Lin, Wenjie Zhang, Yushan Lu, Xinyue Yan, Xiao Jia, Wei Zhang
Abstract
Accurate segmentation of pulmonary lesions is essential for effective clinical diagnosis and treatment strategies. Existing segmentation approaches often lack task-specific semantic guidance, as text-based annotations typically offer coarse localization of lesions, leading to inadequate delineation of lesion boundaries and poor performance on small-scale lesions. To address this, we propose CGSM, a Concept-Guided Segmentation Model that integrates LLM-generated and clinically reviewed concepts into the segmentation process. Specifically, we design a Concept-Visual Alignment Module (CVAM) to activate relevant tokens within the concepts that align with visual features, enhancing the interaction between textual and visual information. In addition, we introduce a Concept Modulated Decoder (CM-Decoder), which uses concepts from CVAM as modulation signals to facilitate the adaptive fusion of image and text features, improving the segmentation accuracy. Extensive experiments on two public datasets show that CGSM achieves state-of-the-art performance, with results of 91.59% Dice and 84.49% mIoU on the QaTa-COV19 dataset, demonstrating its effectiveness in pulmonary lesion segmentation.