DriveUniTok unifies multiple driving data types for better prediction
DriveUniTok: A Unified Multimodal Tokenizer for Driving World-Action Models
Computer Vision and Pattern RecognitionRobotics
Summary
Making self-driving cars work well requires understanding many types of information like camera images, LiDAR (a type of 3D sensor), and the car’s own movements. The authors point out that past methods looked at these separately, which can miss connections between them. They created DriveUniTok, a way to combine all these data sources into one shared code. This helps the car predict what will happen next and plan better actions. Their tests show this unified approach uses less computing power and works better at predicting the future compared to older methods.
What this means in practice
- •For autonomous vehicle engineers: Improve vehicle prediction accuracy and planning efficiency by encoding camera, LiDAR, and motion data into a single representation.
- •For robotics developers: Build better multi-sensor world models for robots navigating complex environments using unified latent representations.
Authors
Jingbo Xu, Hanchao Leng, JiaHui Huang, Hanbing Li, Jingwei Zhao, Kuiyuan Yang, Chen Gao, Guang Chen, Hangjun Ye, Yong Li
Abstract
World models are emerging as a fundamental component of policy learning for real-world agents, enabling agents to predict future world states and plan actions within a learned representation of the dynamic environment. Existing approaches, however, often model appearance, metric geometry represented by projected LiDAR depth, and ego motion either partially or independently, overlooking the fact that these modalities are different projections of a shared underlying world state. This separation can lead to information loss and cross-modal inconsistency, ultimately limiting downstream performance. Motivated by this unified view of the world, we propose DriveUniTok, a unified multimodal tokenizer for driving world-action models (WAMs) that encodes these modalities in a single decodable continuous latent. We freeze DriveUniTok and adapt a pretrained diffusion model, DriveUniWAM, to predict future unified latents; the tokenizer then decodes them into future appearance, projected LiDAR depth, and ego trajectories. Experiments on NAVSIM demonstrate that our unified design achieves better future prediction and planning performance at lower downstream computational cost than separate-stream alternatives. With reinforcement learning post-training, DriveUniWAM achieves a PDMS of 91.54, reaching state-of-the-art performance among the evaluated WAM-based planners.