Self supervised rgb thermal tracking improves performance without extra labels
End-to-End Self-Supervised RGB-T Tracking without Modality Misleading
Computer Vision and Pattern Recognition
Summary
Tracking objects using both regular (RGB) and thermal images helps get better results, especially in tricky conditions. Most methods need lots of labeled data or rely on imperfect labels made beforehand, which can cause problems during training. The authors designed a new method called ESMTrack that learns directly from the initial frame's annotation and unlabeled data in an end-to-end way, without needing extra labels or pre-made guesses. It prevents one type of image from dominating the learning process and adapts to both types of images to improve tracking accuracy.
What this means in practice
- •For security camera system integrators: Build RGB-T tracking systems that do not require manual bounding box labels beyond the first frame, reducing setup costs and improving robustness in dark or obscured scenes.
- •For autonomous vehicle software teams: Improve object tracking by combining RGB and thermal data learned end-to-end without pre-annotation, enabling better detection in diverse weather and lighting.
Authors
Shenglan Li, Rui Yao, Kunyang Sun, Hong Jia, Yong Zhou, Javen Qinfeng Shi, Xinyu Zhang
Abstract
RGB-T object tracking leverages the complementary characteristics of visible and thermal infrared modalities to improve robustness under adverse conditions. Existing supervised methods typically rely on costly modality-aligned bounding box annotations, while most self-supervised approaches follow a two-stage pseudo-labeling paradigm, making tracker training sensitive to pseudo-label quality and preventing joint end-to-end optimization. In this paper, we propose ESMTrack, a fully end-to-end self-supervised RGB-T tracking framework without offline pseudo-label generation or dense frame-level bounding box annotations. Given only the standard initial-frame annotation used in visual tracking, ESMTrack learns discriminative and temporally consistent representations through two complementary objectives: a grounding triplet loss on annotated initial frames and a cross-frame temporal triplet loss on unlabeled search frames, with reliable samples selected by forward-backward consistency. To address modality dominance bias, ESMTrack employs a three-branch architecture consisting of a fusion branch and two unimodal branches for RGB and thermal inputs. We quantify modality contributions using the Average Peak-to-Correlation Energy by measuring response discrepancies between the fusion and unimodal branches. The resulting reliability estimates guide a training-time modality decoupling mechanism that suppresses dominant-modality shortcuts and adaptively weights cross-modal contrastive learning for task-level alignment. Extensive experiments on five RGB-T tracking benchmarks show that ESMTrack achieves competitive state-of-the-art performance, strong cross-dataset generalization, and real-time inference speed. The source code is available at https://github.com/LiShenglana/ESMTrack.