Video models learn to find and predict hidden actions in unlabeled videos
One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine LearningRobotics
Summary
Understanding what actions cause changes in videos is hard because those actions are often not labeled. The authors show that a pretrained video generation model can be adapted to learn both how to predict future frames and to identify the actions between frames without extra labels. This method, called O2LaF, uses one model reused in two ways to learn these hidden actions more effectively. Their approach works well for tasks like navigation and robot manipulation, even when only small amounts of labeled data are available. They also demonstrate that this idea can extend beyond video generation to other predictive representations.
What this means in practice
- •For robotics engineers: Improve robots’ ability to learn and transfer new actions by using pretrained video models to understand visual changes without labeled data.
- •For autonomous vehicle developers: Enhance navigation systems by extracting actionable insights from video data to better predict and respond to environmental changes.
Authors
Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou
Abstract
Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O$^{2}$LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O$^{2}$LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O$^{2}$LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.