DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning
2026-07-27 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors study how to improve robots' ability to follow language instructions by better predicting how scenes change over time. They note that current models struggle to learn physical motion and timing from mostly static training data. Their solution, called DeVA, has separate parts specialized for video prediction and action planning that share information efficiently. This design helps the robot learn faster and better, especially when guided by extra clues about object properties and depth. Tests in simulations and real robots show their approach works well with less data.
robot manipulationvision-language-action modelsvideo generative modelstemporal causalityspatiotemporal priorspolicy learningaffordancedepth perceptionmulti-level feature transfer
Authors
Mengqi Zhang, Sahil Khose, Simar Kareer, Yuchen Song, Unnat Jain, Judy Hoffman
Abstract
Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations. Video generative models offer a promising foundation by encoding rich spatiotemporal priors through future predictions. However, existing Video-Action Models either couple video and action prediction in a shared backbone, making policy adaptation harder to optimize, or under-utilize video information when guiding the action branch. In this work, we introduce DeVA, a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically salient guidance. DeVA transfers representations from multiple video layers to the action expert, enabling rich information exchange while making policy learning more tractable. It further supervises intermediate video features and the action stream with physically salient guidance (affordance/depth). Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.