Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
2026-07-06 • Software Engineering
Software Engineering
AI summaryⓘ
The authors created Cam2Sim, a tool that turns real-world driving video recordings into realistic driving simulation scenarios using the CARLA simulator. It reconstructs roads, cars, and paths from camera data and uses a technique called Gaussian Splatting to make simulated images look more like the original videos. This helps close the gap between simulation visuals and real-world footage, making tests of self-driving systems more accurate. They tested Cam2Sim in an urban driving setting and found it improved the similarity between simulated and real driving behavior.
Simulation-based testingAutonomous drivingCARLA simulatorGaussian SplattingRGB camera dataTrajectory replayOpenStreetMapROSSimulation-to-reality gapUrban driving scenario
Authors
Davide Jannussi, Stefano Carlo Lambertenghi, Constantin Carste, Andrea Stocco
Abstract
Simulation-based testing enables safe and repeatable evaluation of autonomous driving systems, but its effectiveness is limited by the gap between synthetic simulator outputs and real-world camera observations. To address this problem, we present Cam2Sim, a tool that transforms real-world driving recordings into playable CARLA simulation scenarios. Starting from camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets, and augments the reconstructed environment with Gaussian Splatting to render camera observations that resemble the original recording. The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation, CARLA scenario construction, Gaussian Splatting training, trajectory replay, and closed-loop execution with a system under test. We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model, comparing reconstruction quality, image-generation quality, and closed-loop behavior against both a simulation-only baseline and the real-world target. Results show that Gaussian-Splatting-based rendering reduces the visual gap with respect to standard simulator rendering and improves behavioral similarity to the real-world reference runs. The artifact is publicly available at https: //github.com/ast-fortiss-tum/cam2sim, and a screencast showing the tool is available at https://youtu.be/KmZ74l1__lI