TriDifNet improves pedestrian detection with aligned infrared and visible images
TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception
Computer Vision and Pattern Recognition
Summary
Detecting pedestrians using both regular camera images and infrared images can be hard because the two images might not line up perfectly. The authors developed TriDifNet, a computer system that fixes this misalignment, combines the useful parts of both images, and then detects pedestrians more accurately. It does all three tasks together so the whole process works better and helps find people even when the images don’t match up exactly. Tests show it improves detection on a commonly used pedestrian dataset.
What this means in practice
- •For autonomous vehicle developers: Improve pedestrian detection systems by integrating aligned RGB and infrared images for safer navigation in poor lighting or complex environments.$Commercial implications: Enables more reliable pedestrian detection features in self-driving cars by addressing sensor misalignment issues with infrared and visible cameras.
- •For security system designers: Create surveillance systems that can better detect people by combining misaligned infrared and visible camera feeds into a unified detection pipeline.
Authors
Qianchao He, Jing Liu, Yunhao Li, Xiaoqing Tian
Abstract
Infrared--visible (IR--VIS) multi-modal perception supports robust scene understanding under challenging illumination. Infrared and visible images are often misaligned due to platform motion, viewpoint changes, and modality-dependent distortions, causing registration errors to propagate into subsequent fusion and detection stages. We propose TriDifNet, a unified three-task diffusion network for end-to-end RGB-T perception. To the best of our knowledge, TriDifNet is the first diffusion-based network that jointly optimizes registration, fusion, and detection for misaligned RGB-T pedestrian perception. TriDifNet consists of three diffusion-based modules, i.e., the Registration Diffusion Module (RegDM), the Fusion Diffusion Module (FusDM), and the Detection Diffusion Module (DetDM). RegDM aligns the unaligned infrared image to the optical reference by estimating a dense deformation field. FusDM performs residual conditional diffusion to generate a detection-oriented fused representation with optical structural cues and complementary infrared saliency. DetDM formulates pedestrian detection as an iterative denoising process in bounding-box space. To couple the three tasks, TriDifNet introduces a closed-loop optimization mechanism, where detection semantics guide fusion and registration, and the fusion result further regularizes geometric alignment. A stage-wise progressive training strategy and a lightweight Nash multi-task learning (Nash-MTL)-based late-stage refinement are further adopted to balance registration, fusion, and detection optimization. Extensive experiments on the KAIST multispectral pedestrian benchmark show that TriDifNet achieves superior downstream detection performance under misaligned RGB-T settings. Against strong system-level competitors, TriDifNet achieves 65.44\% Recall, 57.55\% mAP$_{50}$, and 26.83\% mAP$_{50:95}$.