Method improves stability of open-vocabulary remote image segmentation
A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images
Computer Vision and Pattern Recognition
Summary
Semantic segmentation means labeling every pixel in an image to identify objects. Remote sensing images often look the same even if rotated or flipped, but existing methods sometimes get confused by these changes. The authors developed a way to use multiple transformed views of an image and fix unreliable parts by comparing them, helping the model stay consistent. They also fine-tune certain parts of the model while it works to better handle changes in the data. Their method works better across many remote sensing image tests.
What this means in practice
- •For remote sensing analysts: Improve the accuracy and consistency of labeling objects in satellite or aerial images despite geometric transformations.
- •For urban planners: Use more stable segmentation results to monitor and assess land use changes in geographic areas from remote sensing data.
Authors
Jianzheng Wang, Huan Ni, Xiaonan Niu, Danfeng Hong, Haiyan Guan
Abstract
The semantic information of objects in remote sensing images is typically invariant to geometric transformations from the dihedral group D4. However, SAM3-based open-vocabulary semantic segmentation (OVSS) methods often exhibit inconsistent responses to different geometric transformations. To exploit this property and improve the stability of OVSS for remote sensing images, we propose a feature-adaptive manifold repair method based on dihedral-group geometric transformations. First, we introduce multi-scale harmonic-guided D4 view selection (MH-D4VS) to select complementary candidate views from a set of geometrically transformed views. Next, we propose original-view-anchored adaptive manifold repair (OAMR), which uses the original view as an anchor and reliable cross-view information to selectively repair locally unreliable visual features. Finally, we develop pixel decoder test-time adaptation (PD-TTA) for SAM3, which fine-tunes only the parameters of the GroupNorm layers online during inference, thereby enhancing the model's ability to adapt to sample-level distribution shifts. Experimental results show that the proposed method achieves an average mIoU of 55.6% across eight remote sensing semantic segmentation benchmarks and delivers consistent performance improvements under different SAM3-based inference frameworks.