C^2GR: Coupled Comprehensive Generative Replay for a Continually Learnable Universal Segmentation Model
2026-06-22 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on improving universal medical image segmentation models that learn new tasks over time without forgetting previous ones. They address the problem of forgetting caused by changes in both image appearance and segmentation goals as new tasks arrive. Their solution, called C²GR, generates realistic image-mask pairs from past tasks to help preserve old knowledge, using advanced methods called Bayesian Joint Diffusion and Relation-aware Unified Prompt Synchronization to keep the generator and segmentor aligned. Their experiments show that their approach reduces forgetting effectively compared to training on all data at once. The authors plan to share their code publicly.
Universal segmentationTask-Incremental LearningCatastrophic forgettingMedical image segmentationGenerative replayBayesian Joint DiffusionRelation-aware promptsConditional denoisingImage-mask correspondenceMulti-task learning
Authors
Wei Li, Jingyang Zhang, Guoan Wang, Junzhi Ning, Yang Chen, Guang Yang, Lixu Gu
Abstract
Universal segmentation models exhibit significant potential for diverse tasks involving different imaging modalities and segmentation objectives. Task-Incremental Learning provides a privacy-preserving approach to continually evolve a universal model on tasks from sequentially-arriving medical departments. However, training the model solely on the incoming task induces forgetting on past tasks, since consecutive tasks exhibit concurrent shifts in image appearance and segmentation objective. To address this problem, we propose a novel Coupled Comprehensive Generative Replay (C^2GR) framework that simultaneously synthesizes image-mask pairs of previous tasks to mitigate forgetting under concurrent appearance and objective shifts. This requires preserving image-mask correspondence for structure-realistic generation and bridging asynchronous optimization of the generator and segmentor for segmentation-oriented generation. Specifically, we propose a Bayesian Joint Diffusion (BJD) method that formulates the correspondence as conditional distributions optimized via conditional denoising. Furthermore, we develop a Relation-aware Unified Prompt Synchronization (RUPS) scheme to simultaneously modulate the generator and segmentor via a shared task-relation-aware prompt for synchronizing their optimization. Experiments on 20 tasks spanning diverse modalities and objectives demonstrate that C^2GR exhibits only a 2.44% drop in overall performance compared to joint training with all task data, effectively alleviating forgetting from the concurrent shifts. Our code will be made publicly available at https://github.com/mar-cry/C2GR.