World latent action modeling improves robot task success from videos
WLA$^3$: World Latent Action Modeling for Semantics, Dynamics, and Kinematics
Robotics
Summary
Training robots to understand and perform actions is hard because labeled action data is limited, especially from human videos. The authors created a method called WLA³ that learns compact representations of how the world changes over short time intervals using video and robot data together. These shared representations help robots understand both the meaning and physical details of actions, enabling better task performance. They show improved success on real robot tasks and can transfer knowledge from human videos to robots.
What this means in practice
- •For robotics engineers: Use shared latent action representations from videos and robot data to improve robot learning and control over multiple manipulation tasks.
- •For computer vision engineers: Develop systems that extract meaningful action information from egocentric videos to support downstream robot behavior models.
Authors
Peidong Liu, Zhiyuan Xiang, Mingyang Li, Wenhao Li, Jiale Zhang, Jiahao Sun, Jiawei Li
Abstract
Scaling generalist policy models with heterogeneous data is limited by the lack of unified, low-noise action supervision. Human egocentric videos are abundant, but only a small fraction comes with high-quality hand-action labels. Observed world transitions offer a common source of action-related supervision across data sources. We introduce WLA$^3$ (World Latent Action Modeling for Semantics, Dynamics, and Kinematics), a unified generalist policy model framework built around representations learned by a World Latent Action Model (WLAM). WLAM first learns how multimodal world states change over a local interval, encoding synchronized camera views and available embodiment-state changes into a compact local latent action and a richer transition feature. Reconstruction from partial modalities and consistency across overlapping windows encourage robust transition representations. WLA$^3$ reuses them across semantics, dynamics, and kinematics: local latent actions support action-sensitive physical-dynamics modeling, segment-level features directly supervise the VLM through a Semantic Latent Aggregate (SLA), and an action expert jointly predicts latent actions together with embodiment-specific robot controls. Human videos provide scalable transition supervision, while robot trajectories ground the shared representation in executable native controls. On LARYBench, the final 32D latent action reaches 67.89\% average classification accuracy. WLA$^3$ achieves 81.9% average success across six real-robot tasks versus 66.2% for $π_{0.5}$. Performance improves as generalist policy model mid-training data scales, and human videos support human-to-robot transfer. Project page can be found at https://wla-3.github.io/.