Adapting features before upsampling boosts image segmentation accuracy
Task Adaptation Through Guidance in Frozen Feature Upsampling
Computer Vision and Pattern Recognition
Summary
Improving computer-generated image segmentation often involves processing features before making pixel-level predictions. The authors show that adapting features before increasing image resolution works better than doing it afterward. They designed tests that kept some classifier outputs the same while allowing spatial attention to improve. Their experiments on common image datasets found small but consistent improvements by focusing on adapting spatial guidance during upsampling.
What this means in practice
- •For computer vision engineers: Implement feature adaptation before upsampling to improve segmentation accuracy in object and scene labeling tasks.
- •For autonomous vehicle developers: Enhance road scene segmentation by adjusting spatial guidance prior to upsampling in vision models for improved environment understanding.
Authors
Pingan Wang
Abstract
Adapting features before upsampling can improve dense prediction, but the gains may arise from changes in feature values, spatial aggregation, or both. We investigate this ambiguity by comparing matched residual adapters before and after a frozen JAFAR upsampler under a shared parent. We construct a readout-null intervention that preserves a frozen linear classifier's coarse logits while allowing spatial attention weights to adapt. Guidance/value exchanges on trained models and retrained fixed-guidance controls test the utility of this adaptation route. In three-seed experiments on PASCAL Visual Object Classes (VOC) and Cityscapes, adaptation before upsampling improves mean intersection over union (mIoU) by 0.807 and 1.413 percentage points, respectively, over adaptation after upsampling. Across both datasets and the JAFAR and AnyUp upsamplers, readout-null adaptation produces mean gains of 1.379-1.987 points over the corresponding frozen linear baselines. Retrained controls further support the benefit of adaptive spatial guidance. These findings explain how dense predictions can improve despite unchanged coarse classifier outputs and identify spatial guidance as a useful target when adapting frozen feature upsampling systems.