Multimodal model routes text and image features to segment medical scans accurately
Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Medical images can be hard to analyze because the areas of interest might be unclear. The authors designed a system called MRSeg that uses both images and written medical descriptions to better identify specific regions in scans. Their method smartly mixes and adapts visual and language information, improving accuracy while keeping the model efficient. Tests on chest CT scan datasets show it segments lung issues more precisely than some existing methods.
What this means in practice
- •For medical imaging developers: Create software that uses clinical text descriptions to improve automated segmentation of lung abnormalities in CT scans.
- •For radiology software teams: Enhance diagnostic tools by integrating adaptable routing of image and text features to better delineate disease regions under complex language instructions.$Commercial implications: Enables more precise clinical imaging products that improve diagnostic segmentation using language guidance, valuable for healthcare providers.
Authors
Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum, Assame Arnob, Tracy Hammond
Abstract
Textual descriptions can reduce ambiguity in medical image segmentation by specifying the finding and location to be delineated. Existing text-guided methods mainly improve where image and language features interact but generally retain a single learned update pathway across all image-text pairs. We propose MRSeg, a parameter-efficient framework that uses each image-text pair to route the adaptation of visual and textual features before dense prediction. Frozen ConvNeXt-Tiny and PubMedBERT encoders provide multiscale visual features and clinical text tokens. A joint router uses the deepest visual feature and pooled text to predict a sparse mixture over low-rank adapter bases. The resulting route is shared across separate adapter banks for two visual scales and text, coordinating their adaptation while keeping the feature-specific parameters separate. Region Bridge uses text-derived queries to aggregate dense visual tokens into latent regions, refines these regions through self-attention and text cross-attention, and redistributes the refined information back to the feature maps. Finally, a multiscale decoder combines refined semantic features with shallow image evidence. On QaTa-COV19 and MosMedData+, MRSeg achieves 90.90/83.32 and 81.53/68.82 Dice/mIoU, respectively, with 7.11M trainable parameters and 7.60 GFLOPs. Code: https://github.com/maklachur/MRSeg.