RoboDream: Compositional World Models for Scalable Robot Data Synthesis
2026-06-01 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new method to generate robot learning data by combining realistic videos of robot actions with different objects and environments. Instead of needing lots of real-life practice, their system reuses robot movements and imagines new scenes, which saves time and effort. They showed that training robots with this generated data helps them learn better and cuts down the need for hard-to-get real demonstrations. Their approach also allows people to control robots without needing physical objects during data collection.
robot learningteleoperationvideo diffusion modelsembodiment hallucinationsworld modeldata generationtrajectory executionscene synthesismanipulation taskspolicy performance
Authors
Junjie Ye, Rong Xue, Basile Van Hoorick, Runhao Li, Harshitha Rajaprakash, Pavel Tokmakov, Muhammad Zubair Irshad, Vitor Guizilini, Yue Wang
Abstract
Scaling robot learning requires large-scale, diverse demonstrations, yet real-world data collection via teleoperation remains prohibitively expensive and time-consuming. While video diffusion models offer a promising avenue for data scaling, existing generative approaches are often limited to superficial visual augmentation, or suffer from embodiment hallucinations that yield physically infeasible motions. We present a generalizable embodiment-centric world model that achieves scalable data generation by synthesizing photorealistic demonstrations with novel objects, in novel scenes, and from novel viewpoints. Our approach anchors generation to rendered robot motion while conditioning on explicit scene and object priors, effectively decoupling trajectory execution from environment synthesis. This formulation has the potential to unlock two powerful data scaling capabilities: (1) retrieval and rebirth, which repurposes existing trajectories into entirely new contexts without new motion data; and (2) prop-free teleoperation, where operators manipulate empty air and the model hallucinates the target objects and scene afterwards, eliminating reset time. We demonstrate with real-world experiments that our generated data consistently improves downstream policy performance and significantly reduces real-world data requirements across diverse manipulation tasks.