World action model generates real time driving and control actions fast
OSWAM: An Effective World Action Model with One-Step Action Generation
Computer Vision and Pattern RecognitionRobotics
Summary
Generating actions for self-driving cars and robots usually takes many steps, which slows them down. The authors found that starting with better guesses about actions helps reduce these steps to just one. They introduced a new method called OSWAM that uses special tokens to give physical hints and learn from future information. This lets the model produce good actions very quickly, making real-time decisions possible. Their tests show OSWAM works as well or better than slower methods for driving and robot control.
What this means in practice
- •For autonomous vehicle engineers: Generate driving control actions in real time with low latency for self-driving cars using an improved one-step action generation model.
- •For robotic control developers: Create efficient embodied control actions for robots that require fast decision making using the OSWAM model.
Authors
Guowen Zhang, Liyi Chen, Xu Hu, Pengfei Wang, Jingfan Chen, Chenhang He, Lei Zhang
Abstract
Direct-policy World Action Models (WAM) have demonstrated the benefits of decoupling action generation from explicit future visual prediction for efficient inference in autonomous driving and embodied control. However, they still require tens of denoising steps to achieve competitive action quality, incurring substantial latency and hindering real-time deployment. Simply reducing the number of denoising steps will collapse the nonlinear multi-step denoising trajectory into coarse updates, substantially degrading action quality. To reduce the denoising steps while enabling real-time action generation, we present an effective world action model with one-step action generation, namely OSWAM, for autonomous driving and embodied control. We argue that it is the absence of structured action priors in random noise initialization that forces direct-policy WAMs to recover coherent action sequences through iterative denoising. Hence, we construct imagination tokens to initialize the diffusion process, providing physical priors and action-conditioned representations. We further introduce privileged information distillation with future conditioning to enhance future-aware action generation. As a result, OSWAM can efficiently and effectively generate high quality actions in just one diffusion step. Our experiments demonstrate that OSWAM achieves comparable or even better results in both autonomous driving and embodied control than direct-policy WAM and Joint-WAM that require tens of steps. It obtains approximately $3\times$ speedup over direct-policy WAM baselines and supports real-time compiled inference with latency as low as 40ms on an NVIDIA RTX 5090 GPU. The project page is available at https://gwenzhang.github.io/OSWAM/