CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
2026-06-02 • Robotics
Robotics
AI summaryⓘ
The authors present CLAW, a system that learns to understand actions and predict their effects from just videos without any labeled actions. CLAW creates a world model and figures out hidden action representations at the same time, helping it learn how actions change the environment. This allows CLAW to copy behaviors by watching videos and plan steps to reach goals without needing action labels. Their tests show CLAW's learned actions make sense and work well for copying and planning tasks. Overall, the authors demonstrate a new way to learn actions and environment dynamics purely from visual data.
self-supervised learningworld modellatent actionsadversarial latent regularizationdiffusion modelsvideo generationimitation learningbehavior cloninggoal-directed planningaction representation
Authors
Tewodros Ayalew, Matthew Jeung, Samuel Wheeler, Xiao Zhang, Andre de la Cruz Arce, Kaylene Stocking, Michael Maire, Matthew R. Walter
Abstract
We introduce CLAW, a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free videos. Our approach leverages adversarial latent regularization and diffusion-based video generation to capture structured and semantically meaningful action representations while modeling rich, predictive environment dynamics, without relying on any action labels or annotations. By simultaneously training the Latent Action Model and world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone. We show that the resulting latent action world model supports both imitation learning from observation and goal-directed planning. In imitation learning, latent actions extracted from raw videos enable behavior cloning. For planning, CLAW generates sequences of latent actions and maps them to executable actions to reach desired goals. Extensive experiments across diverse tasks and embodiments demonstrate that CLAW produces semantically meaningful latent action representations, supports effective action transfer, and enables planning and imitation from observation, outperforming existing methods.