Digital twin helps create real driving data without extra annotations
Digital Twin-Driven Real2Sim2Real: Simulator-Conditioned Generation via Paired Driving-Scene Reconstruction
Computer Vision and Pattern Recognition
Summary
Collecting and labeling real driving images for self-driving cars is expensive and slow. The authors show how using a digital twin—a detailed virtual copy of a real area—allows them to create realistic simulated images that can train 3D object detectors without needing new real images. Their method uses special image generation models conditioned on the simulated scenes inside the digital twin, producing photos that closely match actual driving conditions. Tests show these generated images can nearly replace real data for training, reducing time and cost in new locations.
What this means in practice
- •For self-driving car developers: Train 3D perception systems using synthetic images from a digital twin instead of costly new real-world data collections.
- •For urban planning teams: Create realistic visual data for city-scale simulations and traffic analysis without extensive camera data capture.
Authors
Hojun Lim, Hyeongseok Jeon, Donghyun Kim, Soonyoung Jung, Heecheol Yoo
Abstract
Camera-based 3D perception for autonomous driving relies heavily on large annotated datasets, and deploying such a system to a new target region typically requires data collection and annotation. Generative augmentation has been proposed to reduce this cost, but existing approaches face a fundamental trade-off: label-conditioned methods consume the very annotations they aim to replace, while simulator-conditioned methods offer free annotations but lack visual grounding to specific real environments. This work investigates the extent to which a digital-twin-driven Real2Sim2Real pipeline (DT-R2S2R) can substitute for target-region real data. By reconstructing recorded driving clips inside a georeferenced digital twin (DT-R2S), we condition a diffusion model on geometrically aligned simulator renderings, establishing a digital twin-grounded Sim2Real model (DT-S2R). As a result, DT-S2R synthesizes photorealistic driving images given low-cost yet georeferenced simulator data across both reconstructed and novel simulator scenes within digital-twin coverage. The efficacy of generated data is verified on diverse 3D detectors. DETR3D, especially, reports 93.18% of mAP obtained by a target-region real-data oracle, without employing target images for detector training. Furthermore, simple co-training with existing out-of-target real data outperforms the oracle. Thus, DT-R2S2R can substantially reduce the cost of manual on-site data collection and annotation in digital twin-available districts, providing a practical foundation for scaling 3D perception.