MCSeg: Pre-training and Fine-tuning Volumetric Pyramid Transformer for Multi-modal Cardiac Image Segmentation
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created MCSeg, a new type of computer program that helps automatically identify parts of the heart in 3D medical images from different machines. They fixed a problem by designing a special way to connect two parts of their system so it can better understand both the big picture and small details. They trained their program in two steps: first teaching it to predict missing parts of images, then fine-tuning it to get sharper boundaries. In tests on several heart image datasets, MCSeg worked better than many existing methods, especially when there wasn't much training data.
cardiac image segmentationvolumetric transformer3D Vision TransformerConvolutional Neural NetworkScaling Feature Pyramidself-supervised pre-trainingmasked image modelingregional mutual information lossmulti-modal imagingfew-shot learning
Authors
Zhiyu Ye, Hairong Zheng, Tong Zhang
Abstract
Automatic cardiac image segmentation is pivotal for diagnosing and treating cardiac diseases. In this work, we introduce MCSeg, a volumetric transformer-based network tailored for multi-modal cardiac segmentation. To overcome the architectural mismatch inherent in existing hybrid networks, we propose a novel Scaling Feature Pyramid (SFP). Unlike conventional skip connections, the SFP effectively bridges the single-scale 3D Vision Transformer (ViT) encoder and the multi-scale CNN decoder by transforming the ViT's output into a hierarchical feature pyramid, ensuring that global contextual information is effectively leveraged. For the training paradigm, the ViT encoder first undergoes self-supervised pre-training via masked image modeling. Subsequently, the network is fine-tuned on downstream tasks, during which a regional mutual information (RMI) loss is integrated to improve boundary segmentation accuracy. In experiments, MCSeg consistently outperforms eleven SOTA methods on CT dataset ImageCHD, multi-modal dataset MM-WHS, MRI dataset HVSMR-2.0 and MSD Heart, highlighting the effectiveness of our MCSeg for multi-modal cardiac segmentation tasks. Furthermore, MCSeg's superior performance in few-shot experiment showcases its significant potential in adapting to limited data scenarios. Codes and pre-trained ViT-B weights are open-sourced at https://openi.pcl.ac.cn/OpenMedIA/MCSeg