Z sight unifies robot vision language action with future prediction

$Z$-sight: Unifying Vision-Language-Action and Latent World Modeling

RoboticsComputer Vision and Pattern Recognition

Summary

Robots need to understand what they see and decide what to do next. The authors present Z-sight, a model that helps robots predict future events and actions together using video and language information, without heavy computations of full video frames. It learns from both human and robot videos, improving performance on common robot tasks. This approach makes robots better at planning and acting quickly based on what they perceive.

What this means in practice

  • For robotics engineers: Improve robot control systems by integrating vision, language, and action predictions to achieve faster and more accurate task execution.
  • For autonomous vehicle developers: Enhance prediction of future visual scenarios and control actions in navigation systems using combined vision-language-action models.

Authors

Junjie Ye, Yiqi Zhao, Muchen Xu, Celina Shiyu Wang, Hao Dong, Ruohai Ge, Rong Xue, Benjamin J Kolligs, Aditya Dutt, Sasanka Nagavalli, Nassim A. Samad, Gaurav S. Sukhatme, Jyotirmoy V. Deshmukh, Yue Wang, Yuliang Guo

Abstract

We introduce $Z$-sight, a unified robot foundation model that couples vision-language-action (VLA) and generative latent world modeling. Video-based world action models (WAMs) show the value of joint future-and-action generation for control, yet inherit the heavy cost of video generators. Recent latent-space policies avoid pixel generation, yet typically align intermediate features or rely on frozen target encoders, decoupling future prediction from the VLM's native feature space. $Z$-sight instead predicts the future directly in the visual token space in which its VLM perceives the present. Coupling a foresight expert and a continuous action expert with the VLM in a mixture-of-transformers, $Z$-sight jointly denoises future spatial tokens and actions at inference without pixel reconstruction or external encoders. Trained on human video and robot trajectories, $Z$-sight achieves 81.9% success on SIMPLE (+9.6 points over the strongest baseline) and 73.3% across six Unitree G1 tasks (+20.0 points) at 110 ms per chunk, while remaining competitive on tabletop benchmarks.