Video world model predicts object motion across robot and human bodies
Render the Robot, Generate the World: One Video World Model Across Embodiments
Robotics
Summary
It can be hard for computers to predict what will happen when robots or people move because they have to understand both the body's movements and how the environment changes. The authors created Doppel, a method that uses pictures of how the body is supposed to move along with action plans to help predict what will happen in videos. This approach works for many kinds of bodies, like robot arms or human hands, and helps focus the predictions on how objects move and interact. Training on many different kinds of bodies at once made Doppel better at predicting the future in new situations.
What this means in practice
- •For robotics engineers: Improve robot control systems by predicting object and environment changes from multiple robot types using a shared video prediction model.
- •For augmented reality developers: Generate realistic scene changes based on human or robot pose predictions for interactive AR experiences.
Authors
Yash Jangir, Karan Mirakhor, Renos Zabounidis, Muhammad Zubair Irshad, Sergey Zakharov, Yonatan Bisk, Homanga Bharadhwaj
Abstract
Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.