World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation
2026-07-20 • Robotics
Robotics
AI summaryⓘ
The authors address the problem of making robot simulations match real-world behavior better, especially when some important details are hidden and can't be guessed from what the robot sees. They propose a method called World Translation that works by looking backward at what actually happened, using both imperfect simulators and accurate learned models to understand hidden factors. This approach helps improve robot movement predictions more than previous methods, especially in tricky situations where history doesn't help. They tested it on several robot types and found it worked well, including on a real quadruped robot.
simulation-to-reality gaproboticsdynamics modelingpartial observabilitydomain translationlearned dynamics modelpolicy transferunobservable factorsrobot simulatorsreal-to-sim methods
Authors
Xinchen Yao, Leixin Chang, Hua Chen
Abstract
The gap between simulation and reality remains a fundamental challenge in deploying simulation-trained robotic policies in the real world. Real-to-sim methods narrow this gap from the real side, learning transition dynamics from real data to build a more realistic digital world. Learned dynamics models are their dominant instance. Such methods, however, face a partial observability problem: the same observation may branch to different transitions due to unobservable factors. Existing methods assume these factors can be recovered from observation history. However, this may fail whenever observation history is uninformative, such as a sudden contact event with no prior warning. To address this limitation, we propose \textit{World Translation}, which exploits a complementary strength of simulators and learned dynamics. Simulators are deterministic but physically imperfect, while learned models are accurate but underdetermined under partial observability. Rather than predicting transitions forward from history, we extract the unobservable dynamics information backward from an observed transition, then translate this feature across simulation and reality as an unpaired domain-translation problem that preserves dynamics content while transferring domain style. Experiments across humanoid, quadruped, and manipulator platforms show that our method achieves more accurate dynamics modeling than baselines, with the largest gains when unobservable factors cannot be recovered from observation history. Real-robot deployment on Go2 quadruped confirms improved policy transfer.