Medical image segmentation improved by combining segment anything model and mamba unet
Segment Anything Model Based Context Guided Mamba Unet for Medical Image Segmentation
Computer Vision and Pattern Recognition
Summary
Medical image segmentation helps doctors identify different parts of images like scans or skin photos. The authors found that a popular model called SAM works well in general but not specifically for medical images. To fix this, they combined SAM with another method called Mamba UNet, which is good at understanding details, and added special modules to make the combination work better. This new approach, called SAMBA-UNet, shows improved accuracy on several medical image datasets.
What this means in practice
- •For medical imaging teams: Improve segmentation accuracy for clinical images to aid diagnosis and treatment by integrating global and local image features with domain adaptation.
- •For healthcare ai developers: Build more robust medical segmentation tools by fine-tuning general models on medical data using adapter and LoRA techniques to retain fine anatomical details.
Authors
Chenchen Wang, You Zhou
Abstract
Medical image segmentation is crucial for clinical diagnosis and treatment planning, requiring both global contextual understanding and fine-grained structural precision. While the Segment Anything Model (SAM) demonstrates strong generalization capabilities, its direct application to medical images often lacks domain-specific adaptability. Meanwhile, State Space Models (SSMs)-based methods, such as Mamba, efficiently capture long-range dependencies but may struggle with preserving fine anatomical details. To address these challenges, we propose a segment anything model based context guided mamba unet, SAMBA-UNet, a novel framework that integrates SAM's context knowledge with Mamba's hierarchical feature learning to enhance both global representation and local detail refinement. Specifically, we introduce the Hierarchical Context Enhancement Module (HCEM), an attention-based fusion module that deeply incorporates the multi-level features of SAM and Mamba to improve detail perception and segmentation precision. Additionally, we employ LoRA and Adapter-based fine-tuning to adapt SAM to medical imaging tasks, further enhancing its domain-specific segmentation capability and robustness. We conducted a comprehensive study on several medical segmentation datasets, including Synapse, ISIC17, and ISIC18, evaluating our method across multiple metrics. Experimental results demonstrate that SAMBA-UNet achieves state-of-the-art performance, effectively balancing generalization, segmentation accuracy, and computational efficiency. Our code will be released upon acceptance.