Skeleton guidance improves robot world models for action prediction
Skeleton-Grounded Latent Action Model for Robot World Models
Robotics
Summary
Robot world models help predict what happens next when a robot moves or interacts with objects. The authors show that by using the robot's skeleton (its joint structure) to guide learning, they can separate the robot's own motion from how the environment changes. This makes the model better at imagining future scenes conditioned on specific robot actions. Their approach also lets them transfer human motions to robot movements in these imagined videos and predict task success more accurately.
What this means in practice
- •For robotics engineers: Generate predicted visual outcomes from planned robot actions with improved accuracy by integrating robot skeleton data into simulation models.
- •For animation studios: Create realistic robot motion videos by converting human skeleton movements into robot joint actions without additional training data.
Authors
Qi Lv, Jiayi Lin, Yuxin Jiang, Jianzong Wang, Xiang Deng, Mike Shou Zheng
Abstract
Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.