Improving multi-object tracking by controlling RGB and infrared reliability
Reliability-Consistent Appearance Modeling for RGB-Infrared Multi-Object Tracking
Computer Vision and Pattern Recognition
Summary
Tracking multiple objects using both regular RGB cameras and infrared cameras can be tricky when one type of camera doesn’t work well temporarily. The authors propose a way to keep separate information from each camera type instead of mixing them into one, so the system can decide which one to trust more at any moment. This helps avoid mistakes when objects are identified and keeps better memory of their appearances over time. Their method outperforms previous ones on standard tests, showing how important it is to manage the reliability of different camera inputs.
What this means in practice
- •For security monitoring teams: Enhance tracking accuracy in surveillance systems that use both RGB and infrared cameras by reliably weighting input quality from each sensor type.
- •For autonomous vehicle developers: Improve object tracking in vehicles by dynamically adjusting to varying camera conditions using separate reliability measures for RGB and infrared inputs.
Authors
Bingyang Wang, Jiazuo Yu, Lu Zhang, Lihe Zhang, Huchuan Lu, You He
Abstract
Recent RGB-infrared multi-object tracking (MOT) methods have substantially improved cross-modal representation learning and multimodal information integration. However, most existing methods compress RGB and infrared evidence into a unified descriptor. When one modality is temporarily degraded, the tracker can no longer regulate its contribution separately; unreliable evidence may distort association and contaminate appearance memory. We therefore propose RCA-MOT, a reliability-consistent appearance modeling framework that preserves cross-modal complementarity while retaining post-representation evidence controllability. RGB-specific, infrared-specific, and fused identity representations are maintained as jointly accessible appearance states. For each detection, a simple training-free target-scene estimator produces branch-wise appearance reliability used across the two temporal roles of appearance: relative reliability regulates different identity branches, absolute reliability calibrates borderline candidates near the ReID acceptance boundary, and the same branch-wise reliability controls how strongly matched descriptors update historical appearance memory. The individual components are intentionally simple, focusing evaluation on reliability-controllable appearance organization rather than elaborate fusion or learned reliability-estimation modules. RCA-MOT sets new SOTA results on VT-MOT and UniRTL, improving the previous best HOTA by 5.402/6.075 points and IDF1 by 11.549/12.789 points, respectively, while post-hoc analyses further validate reliability as an informative control signal. Code will be released upon acceptance.