Improving urban LiDAR detection by combining simulated and real data sources
Solution for UCF UrbanTwin V2X-Real Track: Sim-to-Real Urban LiDAR 3D Object Detection
Computer Vision and Pattern Recognition
Summary
Detecting objects with roadside LiDAR in cities can be tricky because simulated data and real-world data differ in many ways, like how dense points are or how pedestrians look. The authors created a system that trains on a mix of simulated and real LiDAR scans, each adjusted to fix specific differences. Their method uses special model parts for each data type and combines their results carefully depending on the object category, such as cars or pedestrians. This approach helps the system better locate and recognize objects when tested on new real-world data.
LiDARsimulation-to-reality gap3D object detectiondigital twinpoint cloudmulti-source trainingclass-aware fusionpedestrian morphologyurban environmentLocalization
Authors
Pu Luo, Cong Xu, Yumei Li, Kexin Zhang, Licheng Jiao, Wenping Ma, Lingling Li
Abstract
Bridging the simulation-to-reality gap in roadside LiDAR requires addressing several coupled discrepancies, including scene geometry, sampling density, return patterns, and pedestrian scale. This report presents a multi-source collaborative training and class-aware fusion framework for Sim2Real 3D detection. The method organizes digital-twin scans, diffusion-redrawn scans, density-stabilized scans, and pedestrian morphology-aligned samples into a unified training pool with complementary roles. Within a common DSVT detection formulation, source-specialized expert branches preserve those roles while optimizing for the same detection objective. At inference, a predefined class-aware fusion pathway integrates geometry-stable and calibration-aware branches for vehicles, sampling-complementary branches for trucks, and morphology-consistent evidence for pedestrians. A label-free point-cloud center blend then refines geometric localization. On the UrbanTwin V2X-Real hidden test set, the unified system achieves a combined score of 0.7421, with 3D mAP@0.5 of 0.4518 and a realism score of 0.8871. The results indicate that a stable, interpretable collaboration among data sources is more valuable than unconstrained aggregation of model outputs.