Grounded-Dreamer generates realistic robot task data for training
Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
Robotics
Summary
Training robots to manipulate objects usually needs a lot of real-world examples, which are hard and costly to get. The authors developed Grounded-Dreamer, a method that creates realistic videos and action data for robots by combining real images with simulated physics. This method filters out impossible actions by checking them in simulation and uses only good examples to teach robots. Their approach avoids needing humans to control the robot during training and helps robots learn tasks from synthetic data.
What this means in practice
- •For robotics engineers: Create large sets of physically valid robot task videos and action data for training manipulation policies without human demonstrations.
- •For automation system developers: Improve robot training speed and task generalization by using visually realistic synthetic data grounded in physics.
Authors
Feng Jiang, Shu Chen, Yuanfei Wang, Yuan Li, Zhenhao Shen, Jiaqi Liang, Yang Chen, Lei Wei, Chen Xie, Ruihai Wu
Abstract
Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.