Real data drives better synthetic scenario generation with faster program search
Real-to-Sim Generation: Synthesizing Scenario Programs from Real-World Data via Constraint Solving
Programming Languages
Summary
Creating fake training data for self-driving cars is hard because computer-made scenes can be unrealistic. The authors present R2SGEN, a method that turns real-world data into structured program descriptions for simulations. Their approach splits the problem into smaller parts and uses smart shortcuts to speed up the search, ensuring the generated scenarios are consistent and realistic. Tests show it works well on complex street scenes and outperforms AI language models at producing plausible programs.
What this means in practice
- •For autonomous vehicle engineers: Create realistic training scenarios from real-world sensor data to improve self-driving system reliability.
- •For simulation platform developers: Integrate faster scenario program synthesis to generate physically plausible simulation environments at scale.
Authors
Peishan Huang, Wenmeng Zhang, Yusen Chen, Zhenbang Chen
Abstract
The demand for synthetic training data is hindered by the sim-to-real gap, as current data-driven and LLM-based generators often produce physically implausible scenarios. To address this, we propose R2SGEN, a Real-to-Sim framework that synthesizes structured scenario programs from real-world data. To overcome the combinatorial explosion and intractability of monolithic Satisfiability Modulo Theories (SMT) encoding, we introduce a decoupled synthesis strategy. This approach separates the discrete structural program search from continuous geometric resolution using lightweight, atomic SMT constraints. Furthermore, we significantly accelerate the search process by integrating two tailored pruning mechanisms: Common Prefix Abstraction-based pruning for Breadth-First Search and Branch-and-Bound for Depth-First Search. We evaluate R2SGEN on 20 real-world scenes of varying complexity from the nuScenes dataset. Experimental results show that our method guarantees consistency with the input scene and produces substantially lower-cost programs than the LLM-based baselines under the evaluated inputs. Both proposed search paradigms exhibit complementary advantages, proving highly efficient and scalable for high-complexity synthetic data generation.