Vision grounded action space improves robot task success rates
ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Robotics
Summary
Robots often struggle to figure out what actions to take based on what they see. This paper introduces ViDAL, a new way to represent robot actions that connects them directly to how the robot’s view of the scene will change after acting. The authors show that this method helps robots perform better on various tasks, succeeding more often in both simulated and real-world experiments. ViDAL also lets robots predict future video of what will happen next, adding an extra sense of understanding.
What this means in practice
- •For roboticists: Implement improved action representations that connect robot motions to future visual changes, raising success rates on multi-task manipulation in robots.
- •For robotics engineers: Integrate a plug-in action interface compatible with existing vision-language-action architectures to enhance robot control and add future scene prediction.
- •For automated warehouse operators: Use vision-dynamics grounded action control to improve reliability and success of robotic arms handling diverse tasks in warehouse environments.$Commercial implications: Enables more accurate and reliable robot manipulation for automated warehouses, improving efficiency and reducing errors in logistics automation.
Authors
Yuan Xu, Yixiang Chen, Qisen Ma, Jiabing Yang, Peiyan Li, Kai Wang, Jianhua Yang, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
Abstract
Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents. However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions. We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene. Specifically, ViDAL learns action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics. When integrated into downstream robot policies, the proposed Action VAE serves as a plug-in action interface compatible with multiple VLA architectures and enables optional future-video prediction as an additional capability. Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task $π_{0.5}$ policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.