SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address a problem in teaching computers to recognize new things in images without forgetting old ones, which is tricky because new learning often erases past knowledge. They propose SELECT, a method that helps the model learn new classes by carefully relating them to similar classes it already knows, instead of treating all past knowledge as the same. Their approach uses a special attention mechanism to transfer context smartly and includes checks to keep the old information intact. Tests on common datasets show their method works better than previous ones at balancing learning new material and remembering old classes.
Class-Incremental Semantic Segmentationcatastrophic forgettingbackground shiftknowledge transferattention mechanismstability-plasticity dilemmaContext Transfer Attentionsemantic segmentationPascal VOCADE20K
Authors
Avi Gupta, Saurabh Yadav, Koteswar Rao Jerripothula, Tammam Tillo
Abstract
Class-Incremental Semantic Segmentation (CISS) is fundamentally challenged by catastrophic forgetting and background shift, where learning new concepts degrades performance on previously seen classes. While existing methods attempt to balance stability (retaining old knowledge) and plasticity (learning new knowledge), they often fail to leverage prior knowledge effectively. These approaches typically rely on indiscriminate knowledge transfer or ambiguous initializations, which can dilute crucial semantic information. To overcome this limitation, we propose SELECT, a novel approach for Selective Context Transfer, which instead grounds each new class in a small set of semantically similar past classes. Its core is a Context Transfer Attention mechanism that aggregates the learned tokens from similar classes into a structured initialization for the new class. To ensure this transfer does not corrupt the borrowed representations, we add a controlled noise perturbation and a margin-based context-transfer loss that enforces separation between the new class token and its source tokens. Extensive experiments on Pascal VOC and ADE20K show that SELECT consistently outperforms prior work, achieving mIoU of 2.2% on VOC and 2.8% on ADE, providing an effective handle on the stability-plasticity dilemma. Code is available at https://github.com/avigupta2798/SELECT.