RLG-TPV: Radar- and LiDAR-Guided Tri-Perspective View Fusion for Camera-Radar 3D Object Detection
Computer Vision and Pattern Recognition
Summary
The authors introduce RLG-TPV, a method that uses both camera images and radar data to better detect 3D objects around a vehicle. Unlike older approaches relying only on cameras, their system uses radar and LiDAR during training to improve understanding of object depth and position. They use a special way to combine features over time using radar velocity to help track moving objects more accurately. The method works well on a popular dataset, improving accuracy and reducing errors compared to previous methods. Their experiments show that combining this geometric guidance from different sensors is key to the improved results.
Authors
Ahmet Mete Dokgoz, A. Enes Doruk, Hasan F. Ates
Abstract
Tri-Perspective View (TPV) representations describe 3D scene structure through top, side, and front feature planes, but existing TPV lifting is primarily camera-based, leaving the depth of sampled image evidence ambiguous along projected camera rays. We propose RLG-TPV, a multimodal TPV framework for camera-radar 3D object detection in which radar and training-time LiDAR provide complementary geometric guidance during representation construction. A ray-guided deformable-attention lift weights sampled image features using LiDAR-supervised camera depth probabilities and radar frustum occupancy, while radar additionally refines the depth distribution before lifting. Because conventional radar provides limited elevation information, LiDAR-derived class-occupancy targets supervise the side and front planes during training; the corresponding heads are removed at inference, so deployment requires only cameras and radar. For temporal aggregation, Doppler-guided temporal fusion aligns past features using a motion field anchored by measured radar radial velocity, with gating that limits warping in regions without supported motion. An RCS-aware radar scatter further allows radar evidence to spread over spatial neighborhoods conditioned on radar cross section. On the nuScenes validation set, RLG-TPV achieves 0.4981 mAP and 0.5959 NDS, reducing orientation and velocity error by 31.9\% and 30.7\% relative to the published CRN baseline. Ablation studies show that ray-level geometric guidance is a major contributor to the final performance.