PhysWAM predicts driving scenes and vehicle actions using 3D geometry
PhysWAM: Physically Consistent World Action Model for Autonomous Driving
RoboticsComputer Vision and Pattern Recognition
Summary
Predicting how a self-driving car should move while also understanding how the world around it changes is hard. The authors created PhysWAM, a model that learns to predict video, depth, and the car’s own movement all at once. By making sure the predicted depth and motion match real 3D measurements from sensors, their model stays physically consistent. This approach helps PhysWAM plan better car trajectories and generate more accurate depth and video predictions, even in new driving environments.
What this means in practice
- •For autonomous vehicle engineers: Generate physically consistent predictions of vehicle movement and scene changes to improve trajectory planning in self-driving cars.
- •For robotics perception teams: Create temporally coherent depth and video forecasts from multiple camera views using geometric constraints for improved robot navigation.
Authors
Dhruv Parikh, Fengcheng Yu, Quankai Gao, Jiawei Yang, Junjie Ye, Maulik Bhatt, Thang Vu, Charles Ochoa, Rowan McAllister, Igor Vasiljevic, Rajgopal Kannan, Viktor Prasanna, Vitor Guizilini, Yue Wang
Abstract
World-action models (WAMs) jointly predict how a scene will evolve and how an agent should act, however joint generation alone does not necessarily impose a shared geometric constraint on these predictions. We present PhysWAM, a unified world-action model for autonomous driving that co-denoises multiview video, metric depth, and ego motion within a single flow-matching transformer. To ground world and action generation in measured scene geometry, we introduce Coupled Point Projection (CPP) that unprojects the generated depth into 3D points, transforms them using the generated $\mathrm{SE}(3)$ ego motion, and minimizes their distance to LiDAR points transformed using the recorded ego motion. This geometric constraint promotes physical consistency with the measured scene by jointly supervising generated depth and motion alongside their standard flow-matching objectives. At inference, trajectory selection relies only on a simple label-free consensus rule, with no learned scorer or simulator feedback. We evaluate PhysWAM across NAVSIM v1 and v2 planning, zero-shot closed-loop transfer, and future video and metric-depth prediction. Despite PhysWAM's simple selection procedure, it achieves strong planning performance and transfers zero-shot to unseen driving environments. It also generates accurate metric depth and temporally coherent video, with CPP improving both planning and depth prediction. Together, these results demonstrate that the geometric relationship between scene depth and ego motion provides a direct way to couple world and action generation within a simple unified model.