Cross-scale sparse attention improves remote sensing image segmentation
Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing
Computer Vision and Pattern Recognition
Summary
High-resolution satellite images have lots of details but are hard to analyze when describing specific areas using natural language. The authors propose a new method called CSSAM that better connects multiple image scales with complex language by focusing only on the most important visual and text information. They also introduce a new dataset to test how well models understand multiple targets, relationships, and different detail levels in descriptions. Their experiments show this approach works well for accurately locating and reasoning about complex areas in satellite images.
What this means in practice
- •For remote sensing analysts: Segment specific regions in high-resolution satellite images based on complex natural language descriptions for detailed scene analysis.
- •For geospatial data platform developers: Integrate advanced multimodal attention to improve accuracy and efficiency in referencing and segmenting targets described by text in RS image platforms.
Authors
Yuxi Li, Lu Si, Xia Liao, Yidong Shen, Zheng Wei, Jun Zhang, Xin Yuan
Abstract
High-resolution remote sensing (RS) imagery provides fine spatial detail over large areas, making it essential for fine-grained scene understanding. Referring Remote Sensing Image Segmentation (RRSIS) aims to segment regions described by natural language, yet many existing methods rely on sequential alignment--fusion pipelines with limited joint modeling of cross-scale visual hierarchies and linguistic dependencies. Moreover, dense attention mechanisms struggle with the sparsity and wide spatial dispersion of RS targets, while current datasets offer limited structured evaluation of multi-target references, multi-hop relational reasoning, and multi-granularity descriptions, limiting both model development and evaluation. To address these challenges, we propose a Cross-Scale Sparse Attention Model (CSSAM), which aligns multiscale visual features with complex referring expressions through two key modules. The Cross-scale Modality-aligned Fusion Module (CMFM) performs unified cross-scale cross-modal fusion, while the Dynamic Multimodal Sparse Hybrid Attention (DMSHA) selectively attends to informative visual--textual tokens, reducing dense visual interactions. Furthermore, we introduce M$^{3}$RRSIS-D, a RRSIS dataset explicitly designed around three core reasoning dimensions: multi-target (M$^{1}$), reference-dependent multi-hop (M$^{2}$), and multi-granularity (M$^{3}$) expressions. Extensive experiments on M$^{3}$RRSIS-D and two public RRSIS benchmarks demonstrate that CSSAM achieves competitive performance across accuracy, fine-grained reasoning, and high-IoU localization metrics, validating its effectiveness for complex semantic reasoning in realistic RS scenarios. Code, dataset, and model weights will be released upon acceptance.