Radar improves moving object tracking in difficult visual conditions
Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Tracking moving objects in videos can be hard when lighting is bad or objects are hidden. The paper introduces a system that combines regular camera images, thermal images, and radar data to better detect and follow moving things. Radar helps by directly measuring speed, which works even when cameras struggle. The authors created a new dataset and a tracking method called SAM-Radar that uses all these data sources together, improving accuracy in tough surveillance scenes. Their approach helps keep track of objects even when they disappear briefly or are hard to see.
moving-object segmentationradar sensingmultimodal data fusiontrackinginstance masksoptical flowthermal imagingMOTAHOTAIDF1
Authors
Jue Wang, Xuan Wang, Hao Zhou, Ruixiang Zhou, Yixuan Zhou, Tianshuo Yuan, Jieming Ma, Jie Zhang, Fei Luo
Abstract
Moving-object perception must decide which image regions correspond to real motion and keep every instance identified over time. Methods that read motion from appearance, optical flow, or estimated trajectories lose that evidence under poor illumination, adverse weather, reflections, and occlusion. Radar is a natural remedy because it measures radial velocity directly instead of inferring it from photometric correspondence. However, existing benchmarks do not jointly provide radar measurements, dense moving-instance masks, and temporally consistent identities for surveillance. We therefore introduce RGBTR-Motion, a synchronized and calibrated fixed-camera benchmark that pairs RGB, thermal, and radar streams with dense instance masks and temporally consistent identities across diverse surveillance scenes. We also develop SAM-Radar, an RGB, thermal, and radar-based segmentation and tracking framework built on SAM 3. SAM-Radar's radar-aware detector fuses calibrated RGBT features with radar returns that are grounded at their projected image locations, and motion supervision, implemented as foreground classification of those projected returns, teaches the detector to reject clutter without any text prompt. The tracker associates accepted radar returns with individual trajectories and uses them as physical evidence that a visually degraded target remains present. This allows it to bridge short periods of low visibility or occlusion and reconnect a reappearing target to its existing identity instead of starting a new track. SAM-Radar attains 0.7027 IoU and 0.8090 F1-50, and raises MOTA, HOTA, and IDF1 by 0.2977, 0.1603, and 0.2857 over the strongest competing values.