Semantic guided method improves object detection in misaligned thermal and RGB images
S2A:Semantic-to-Spatial Alignment for Alignment-Free RGB-T Salient Object Detection
Computer Vision and Pattern Recognition
Summary
Detecting important objects in pairs of regular and thermal images that don't line up well can be tricky because the mismatch mixes up the data. The authors created a method called S2A that first uses overall meaning to connect the two image types and then corrects local misalignments. This approach helps to reduce confusion caused by misalignment without needing to carefully line up the images first. Their method performs well on several tests, showing it can find key objects effectively even when the two images are not perfectly matched.
What this means in practice
- •For surveillance system designers: Improve detection of important objects in mixed visual and thermal camera feeds without expensive image alignment steps.
- •For automotive engineering teams: Enhance object detection in driver assistance systems using unaligned thermal and RGB sensors for safer vehicle perception.
Authors
Qiangqiang Zhou, Yang Luo, Yong Chen, Jiawei Xu
Abstract
Alignment-free RGB-T salient object detection (RGB-T SOD) aims to identify salient objects from unregistered RGB and thermal image pairs without costly pre-alignment. However, spatial misalignment breaks pixel-wise correspondence and causes feature contamination during cross-modal fusion. To address this issue, we propose S2A, a semantic-to-spatial alignment framework for alignment-free RGB-T SOD. Specifically, a global-guided hierarchical fusion module (GGHF) first exploits global semantic guidance to suppress background interference and refine hierarchical intra-modal features. Subsequently, the alignment-free cross-modal channel attention module (AFCA) globally exchanges complementary semantic information through channel-wise interaction, effectively overcoming the interference caused by local spatial misalignments. Finally, a spatial deformable cross-attention module (SDCA) predicts adaptive sampling offsets to recover local cross-modal spatial correspondence. Through this semantic-to-spatial paradigm, S2A first enables reliable cross-modal semantic interaction and subsequently performs local spatial calibration, effectively reducing misalignment-induced feature contamination. Without bells and whistles, S2A achieves highly competitive performance on multiple public alignment-free RGB-T benchmarks, demonstrating its effectiveness in alleviating misalignment-induced feature contamination.