Token-world models robot environments with compact vision language tokens
Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation
Robotics
Summary
Many robot systems simulate future scenes by predicting pictures and then using those pictures to plan actions, which can be slow and indirect. The authors study a new way to simulate by directly working with small, efficient coded pieces of vision-language information instead of images. Their method, called Token-World, makes predictions about future robot states in this compressed form and then uses these predictions to guide robot actions more accurately and quickly. They showed that this approach leads to better long-term predictions and more reliable robot performance in manipulation tasks.
What this means in practice
- •For robotics engineers: Simulate robot manipulation tasks more efficiently by predicting future states in a compressed vision-language token space, improving policy action reliability and speed.
- •For automation system developers: Integrate compact token-based world models into automation pipelines to achieve faster and more accurate decision making for robot control tasks.
Authors
Chuyao Fu, Xiaowei Chi, Yuhan Rui, Yu-kai Wang, Zezhong Qian, Xiaojie Zhang, Yunfan Lou, Kevin Zhang, Kuangzhi Ge, Chak Wing Mak, Zhiyang Chen, Athena Zhuoming Zhong, Hongyang Chen, Haoran Li, Yike Guo, Sirui Han, Shanghang Zhang
Abstract
A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World ($r=0.794$ vs.\ $0.583$), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.