Vision language model improves robot action using physical dynamics
PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots often struggle to understand how things move and behave physically when following instructions and seeing their surroundings. The authors created PDRVLA, a model that learns physical dynamics from vision and language data without needing to generate future videos in real time. This model predicts future visual features like colors, shapes, and movements, helping robots act more accurately and quickly. They trained it using existing public data enhanced with human-like reasoning steps, achieving strong results across several simulated robot tasks.
What this means in practice
- •For robotics engineers: Improve robot control systems by integrating physical dynamics representations without costly real-time video prediction.
- •For automation system developers: Develop robots capable of following complex language instructions and reacting accurately to visual changes by using lightweight physical token models.
Authors
Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang, Yilin Chen, Xu Jiang, Kaiquan Chen, Chongyang Wu, Haoran Li, Fan Yang, Mingan Lin, Wentao Zhang, Zenan Zhou, Youzheng Wu
Abstract
Vision--Language--Action (VLA) models generate robot actions from visual observations and language instructions, but their understanding of physical dynamics remains limited. Video world models can capture scene dynamics, yet generating future videos introduces high latency into the control loop. We introduce PDRVLA, whose lightweight Physical Dynamics Module extracts physical tokens from vision--language features to condition action generation without online video generation. The module learns through direct prediction of future RGB, depth, and optical-flow features, with auxiliary supervision through a frozen pretrained video world model. Training relies entirely on publicly available data, expanded through human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation. Across four simulation benchmarks, PDRVLA achieves success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin~2.0, and 81.5% on SimplerEnv. Ablations show improved future prediction with world-model supervision and evaluate how subtask conditioning, future object-pose supervision, and ECoT training affect action success.