World Tokens: Enhancing Embodied Policies with Training-Time World Modeling
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present World Tokens, a new method for teaching robots or AI agents how to act by combining understanding of images and language with knowledge about how scenes change over time. Their approach uses a special component called a World Adapter that helps the system learn from videos during training but does not slow down the robot when it is actually working. By doing this, their model can predict actions quickly and accurately without needing heavy video processing while running. They tested this method on several benchmarks and real-world tasks, showing it works well even without extra training specifically on robot actions.
vision-language-action modelsworld-action modelsworld dynamics modelingfuture video denoisingembodied policyclosed-loop controlspatiotemporal evolutionvideo world modelsrobotic action predictionfine-tuning
Authors
Qu Tang, Benhui Zhuang, Bo Yuan, Xue Yu, Longteng Guo, Junlan Feng
Abstract
Vision-language-action (VLA) models are a widely adopted paradigm for embodied policies. They excel at efficient closed-loop control but do not explicitly model how physical scenes evolve as a task unfolds. Recently emerging world-action models (WAMs) leverage pretrained video world models to capture spatiotemporal evolution, yet retaining future generation or a large video backbone in the control loop substantially increases inference cost. We introduce World Tokens, an embodied policy architecture built around a World Adapter that bridges visual-language understanding, world-dynamics modeling, and action generation. It uses world modeling during training to enhance the action policy while preserving efficient deployment. Specifically, the World Adapter transforms VLM features into a fixed set of world tokens, which condition a jointly fine-tuned future-video denoiser and simultaneously serve as the action expert's sole visual-language context. This shared conditioning allows gradients from future-video denoising to directly shape the representation used for action prediction, while exclusive routing prevents the policy from bypassing that representation. At deployment, the world-model branch is removed, leaving only the VLM, World Adapter, and action expert, with no online video-model inference. With a 2B backbone and no embodied action pretraining, World Tokens is highly competitive on LIBERO, attains the best reported averages on SIMPLER, substantially improves real-world R1 Pro success over a matched action-only baseline, and generates each action chunk at VLA-level latency.