Automated system converts real workspaces into robot training data
AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning
Robotics
Summary
Training robots to manipulate objects usually requires lots of time-consuming work to gather examples from the real world. The authors present AutoRSR, a system that automatically scans a real workspace and creates digital versions of objects and scenes ready for robot learning without manual effort. It generates robot action examples by starting from a few demonstration motions and scales these with automatic adjustments. Using AutoRSR, robot learning policies trained on this generated data performed similarly to those trained on much more expensive human demonstrations, while saving a lot of human time.
What this means in practice
- •For robotics engineers: Produce large-scale robot training datasets without manual scene calibration or object annotation by scanning real-world workspaces.
- •For manufacturing automation teams: Set up scalable training pipelines for robot manipulators in factory environments by converting physical work areas into ready-to-use simulation assets.
Authors
Jiawei Zhang, Chang Liu, Litao Deng, Tao Zhang, Qin Jin
Abstract
Scaling embodied robot learning requires large and diverse manipulation datasets, but collecting real-world demonstrations remains expensive in human effort and robot time. Simulation provides a scalable alternative, yet converting a real workspace into a robot-learning-ready environment still requires substantial manual effort for object extraction, asset processing, coordinate calibration, and trajectory specification. We present AutoRSR, a highly automated real-to-sim-to real framework that converts scanned real-world workspaces into reusable embodied simulation assets and scalable manipulation data. Given multi-view observations, robot states, and open-vocabulary semantic queries, AutoRSR reconstructs the workspace, extracts individual object assets, and automatically registers the reconstructed scene to the robot base frame at metric scale. Starting from a small number of reference motions, AutoRSR then generates executable manipulation trajectories and scales them through trajectory interpolation and domain randomization. After workspace scanning and semantic queries, the remaining pipeline requires no manual object annotation, scene-to-robot calibration, or robot teleoperation. Across three real-world manipulation tasks, policies trained on 1K automatically generated episodes per task achieve 73.3--86.7% real-world success rates, comparable to pi 0.5 trained from 150 human teleoperated demonstrations, while reducing the human time required for data generation by an order of magnitude. These results demonstrate that real-world observations can be converted into metrically aligned, executable, and scalable simulation data with substantially reduced human effort.