Vision language action policies improve long horizon control with predictive world models
PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies
Robotics
Summary
Controlling robots or AI systems over a long time can be hard because they need to predict what will happen next. The authors developed a method called PLaW-VLA that helps these systems predict important future events without worrying about every tiny visual detail. This approach uses a special model to focus on relevant future states and uses that information to decide the best actions. Experiments show this method helps robots perform better, especially in tough tasks and when conditions change.
What this means in practice
- •For robotics engineers: Improve robot control in complex tasks by using predictive models that focus on task-relevant future states instead of detailed visuals.
- •For autonomous systems developers: Create control policies for autonomous agents that generalize better to new environments by using prediction in latent task-relevant spaces.
Authors
Yu Liu, Hetian Guo, Tianlv Huang, Ziyi Cai, Wudi Chen, Hantang Wang, Qiutong Liu, Yingzhi Peng, Wei Han, Peijun Tang, Jianan Wang, Zipei Fan, Zhiyuan Zha, Xuan Song
Abstract
Learning to predict how the world evolves can provide vision-language-action (VLA) policies with predictive context for long-horizon control, but its effectiveness depends on what future representation is modeled and how it conditions action generation. We introduce PLaW-VLA, which models task-relevant future states in a pretrained prediction-oriented representation space, reducing the need to predict control-irrelevant visual details. Built on a Mixture-of-Transformers architecture, PLaW-VLA conditions action generation on observation history, current task semantics, and predicted future states through structured causal attention. Experiments show a +11.8 percentage-point (pp) gain over reactive policies on RoboTwin Hard Horizon III and a +1.77 pp gain over reconstruction-oriented latent prediction on zero-shot LIBERO-Plus, supporting improved long-horizon control and generalization under distribution shift, respectively. By avoiding low-level visual reconstruction, PLaW-VLA lowers the burden of future prediction, enabling a lightweight latent world model with parallel future prediction and about 1/19 the inference latency of generative world-action modeling at comparable policy performance.