Brain tumor segmentation improved with convolutional experts added to vision transformer
BaT-SAM: Introducing Inductive Bias into SAM via a Mixture of Convolutional Experts for Brain Tumor Segmentation
Computer Vision and Pattern Recognition
Summary
Accurately outlining brain tumors in MRI images is important but slow when done by hand. The authors introduce BaT-SAM, a method that upgrades an existing image analysis AI by adding small special filters to better recognize shapes in the images. This lets the AI automatically find and mark tumors without needing extra hints, using fewer new settings to learn. Their method outperforms previous approaches on standard tests and works well on different sets of images.
What this means in practice
- •For medical imaging software teams: Develop automated MRI analysis tools that locate and segment brain tumors without manual input using efficient fine-tuning on vision transformer models.
- •For radiology workflow integrators: Integrate automatic brain tumor segmentation modules to speed up diagnosis and treatment planning by reducing manual image annotation requirement.
Authors
Sumiaya Jannat, Md Mahedi Hasan, Md Shamimul Islam, Md Nur Hossain
Abstract
Accurate brain tumor segmentation from magnetic resonance imaging (MRI) is essential for diagnosis and treatment planning; however, manual delineation is labor-intensive and time-consuming. Existing approaches are either task-specific with limited generalization or rely on user-provided prompts during inference. We present BaT-SAM, a parameter-efficient adaptation framework that enables fully automatic brain tumor segmentation with the Segment Anything Model (SAM). BaT-SAM injects local inductive bias into SAM's frozen Vision Transformer (ViT) encoder via Conv-LoRA, where image features are processed by a sparsely gated mixture of multi-scale convolutional experts. This design compensates for the limited translation-equivariant inductive bias of vanilla ViTs while fine-tuning fewer than 5M parameters (less than 1% of the full model). To remove the need for user prompts, we further introduce a lightweight self-prompting box regressor that is jointly optimized with the adapter using scheduled sampling and predicts tumor locations directly from image embeddings. On the BRISC2025 benchmark, BaT-SAM achieves a weighted mean IoU of 0.832 while outperforming state-of-the-art benchmarks. It further demonstrates strong cross-dataset generalization, achieving Dice scores of 0.702 on Figshare and 0.784 on the LGG MRI Segmentation dataset.