Summary
Testing self-driving cars often uses computer simulations because testing in the real world can be difficult or unsafe. But simulations don't always match reality, which can lead to incorrect conclusions about how well these systems work. The authors looked at over 200 studies to see how researchers measure and fix the differences between simulation and reality. They found most work focuses on making scenarios and perceptions more realistic but that many studies lack clear evidence linking simulation problems to actual safety impacts. Their analysis suggests tests should clearly state which real-world differences matter and report on specific parts of the self-driving system.
What this means in practice
- •For automotive testing teams: Improve testing workflows by focusing on which real-world differences affect simulation reliability for automated driving systems.
- •For simulation software developers: Develop better simulation tools by targeting components like scenario and actor realism revealed as critical in measured reality gaps.
A survey. It maps existing work.
Authors
Stefano Carlo Lambertenghi, Oliver Weissl, Johannes Betz, Andrea Stocco
Abstract
Simulation is essential for testing automated driving systems (ADS), but discrepancies between simulation and reality can undermine testing conclusions. It remains unclear which discrepancies the literature addresses and what evidence supports correspondence claims. We present a systematic mapping of 235 primary studies that measure or mitigate the reality gap in ADS testing, selected from 4,195 publications. We characterise objectives, targets, approaches, experimental setups, and evaluation evidence. Our classification distinguishes simulated and real-world execution across six ADS test components, separately for each approach and its evaluation reference. This distinguishes real-world information used to construct an approach from evidence used to evaluate its claims. Of the included studies, 76% address mitigation, primarily targeting scenario and actor realism and perception, while 94% report validation or measurement procedures. Only 61 studies report metrics from multiple families covering sensor/perception correspondence, ADS behaviour, vehicle dynamics, or safety outcomes; this coverage does not establish causal links between discrepancies and test verdicts. The synthesis highlights that real-world content and physical execution must be interpreted against the claim evaluated. The findings motivate claim-specific evaluation and component-level reporting to establish which discrepancies affect testing conclusions and when simulation evidence supports conclusions about real-world ADS behaviour.