Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation
2026-06-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of segmenting specific objects in aerial images based on natural language descriptions, which normally requires heavy model fine-tuning. They introduce a new method called S4ECA that efficiently adapts parts of a model to understand both language and images without retraining everything. Their approach uses special adapters for text and images to better connect language meaning with visual details at multiple scales. This results in more accurate segmentation while updating only a small portion of the model's parameters. They tested their method on standard datasets and showed it works well and is computationally efficient.
Referring Remote Sensing Image SegmentationParameter-Efficient TuningCross-modal AlignmentDual-encoder AdapterNatural Language ProcessingMulti-scale Feature ExtractionAerial ImageryFine-tuningSemantic EmbeddingsVisual-Language Models
Authors
Kun Li, Shengxi Gui, Francesco Nex, Michael Ying Yang
Abstract
Referring Remote Sensing Image Segmentation (RRSIS) seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image. While existing RRSIS models have benefited from large-scale foundation models, they predominantly rely on full fine-tuning. These approaches are computationally intensive and may weaken the generalization ability of pre-trained models, as extensive fine-tuning on significantly smaller downstream datasets can distort the well-structured feature representations learned during large-scale pre-training. Although Parameter-Efficient Tuning (PET) offers a potential alternative, existing PET frameworks primarily focus on single-modal optimization, failing to capture the complex cross-modal dependencies required for multimodal reasoning, while simultaneously struggling to bridge the substantial domain gap between natural scenes and aerial imagery. To address these limitations, we propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment (S4ECA), which enables effective and efficient cross-modal interaction through parameter-efficient adaptation. Specifically, we design a dual-encoder adapter architecture. The textual adapter employs learnable queries to distill highly semantic language proxies from word-level embeddings, facilitating early grounding. Simultaneously, the visual adapter refines hierarchical feature representations through a multi-scale dense extractor, followed by a language-guided scale and spatial selection mechanism that dynamically emphasizes relevant visual contexts, ensuring precise cross-modal alignment. By updating only 2.4% of the backbone parameters, our proposed model achieves state-of-the-art performance on the RRSIS-D and RefSegRS datasets, demonstrating superior efficiency and precision in complex aerial scenarios.