RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
2026-06-11 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present RepWAM, a new way to help robots understand and predict their actions by focusing on meaningful visual and action representations instead of just reconstructing images. They create a special tokenizer that links visual inputs with related action tokens, allowing the model to better learn how to follow instructions and predict future states. Their approach improves robot manipulation in both real tasks and simulations, showing benefits over older methods that focus mainly on image reconstruction. This work suggests that focusing on semantic action representations helps build better models for controlling robots.
World Action ModelVisual-Action TokenizerRepresentation LearningRobot ManipulationSemantic Latent SpaceFuture State PredictionClosed-Loop ControlInstruction FollowingVideo TokenizationLatent Actions
Authors
Junke Wang, Qihang Zhang, Shuai Yang, Yiming Luo, Yujun Shen, Zuxuan Wu, Yu-Gang Jiang, Yinghao Xu
Abstract
This work presents RepWAM, a representation-centric world action model (WAM) built on representation visual-action tokenizers. Existing WAMs typically inherit reconstruction-oriented video tokenizers from pretrained video generation models. Although these tokenizers preserve visual fidelity, pixel reconstruction alone provides limited guidance for learning instruction-following dynamics that connect future prediction with robot control. To address this, we explore a semantic visual-action latent space for representation-centric world action modeling. Specifically, we train a representation visual-action tokenizer that maps visual inputs into aligned visual and latent action tokens. We then pretrain our WAM to jointly model future visual states and the latent actions that connect them under language instructions, followed by adaptation to real robot trajectories for closed-loop manipulation. Experiments on real-world manipulation tasks and simulation benchmarks show that RepWAM delivers strong performance across diverse manipulation settings, while ablations highlight the value of semantic visual-action tokenization over reconstruction-oriented alternatives. These results establish representation visual-action tokenization as a promising foundation for world action models and a step toward generalist robot policies. Code and weights will be available at https://github.com/wdrink/RepWAM.