Semi supervised visible infrared object detection with aligned consensus teacher
Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery
Computer Vision and Pattern Recognition
Summary
Detecting objects in images taken by both regular cameras and infrared cameras usually needs lots of detailed labels, which are expensive to get. The authors developed a method called Aligned Consensus Teacher that uses only a small amount of labeled image pairs and many unlabeled pairs to learn accurately. Their approach smartly aligns objects between visible and infrared images and improves predictions by combining knowledge from both types of images. This method works well even when only a small fraction of the image pairs are labeled, achieving performance close to fully labeled training.
What this means in practice
- •For drone surveillance teams: Improve detection of vehicles and objects in paired visible and infrared drone images using fewer labeled examples.
- •For security monitoring operators: Enhance object detection in multi-sensor security cameras by reducing the need for expensive dual-modality labeling.
Authors
Qi Ming, Xiaxin Yuan, Jiahuan Zhou, Jiangmeng Li, Xudong Zhao, Zhanchao Huang, Juan Fang, Shaoguang Huang, Aleksandra Pizurica
Abstract
Visible-infrared object detection (VIOD) detects objects with oriented bounding boxes from paired visible and infrared images. Existing methods depend on costly dual-modality annotations. Semi-supervised learning can reduce this burden, but extending it from single-modal detection to VIOD is challenging. In the practical image-pair-level setting considered here, only a few pairs are labeled in both modalities, while the rest are completely unlabeled. This limited supervision creates three challenges: (i) too few labeled boxes for robust cross-modal alignment; (ii) pseudo-label errors caused by branch-wise misses accumulate during self-training; and (iii) tail-class annotations become critically scarce as the labeling budget decreases. We propose Aligned Consensus Teacher (ACT) for label-efficient VIOD in this setting. Its Cycle-Consistent Region Alignment (CRA) combines cycle consistency and sparse anchors with reliability-weighted regional matching. Cross-Modal Consensus Mean-Teacher (CMC-MT) forms consensus pseudo labels under pair-preserving views to recover branch-wise misses and supervise unlabeled pairs. Text-Guided Cross-Modal Instance Augmentation (TG-CMIA) uses a vision-language scene prior to compose tail-class instance pairs while preserving RGB--IR offsets. To the best of our knowledge, ACT is the first framework to study semi-supervised VIOD under this image-pair-level setting. Experiments on DroneVehicle and VEDAI show consistent gains across annotation ratios. With 10\% labeled pairs on DroneVehicle, ACT reaches 94.3\% of the mAP obtained by the same detector under full supervision. Code and models will be available on GitHub to facilitate future work.