DiffWAM enables fast efficient navigation for drones from video features
DiffWAM: A Fast and Efficient Navigation World Action Model
RoboticsArtificial Intelligence
Summary
Flying drones through complex environments usually needs detailed video predictions and 3D maps, which take a lot of computing power. The authors present DiffWAM, a method that uses frozen video models to directly turn visual features into smooth drone flight paths without expensive steps like future video generation. Their system works in real time on hardware suited for drones, successfully guiding them through tricky maneuvers. This shows it’s possible to control drones efficiently using smart video understanding.
What this means in practice
- •For drone developers: Create on-board controllers that synthesize continuous 3D flight paths from video inputs without costly video generation steps, improving real-time navigation.
- •For robotic system integrators: Integrate fast, geometry-aware motion prediction models into robots performing complex navigation tasks like orbiting or constrained traversal.
Authors
Mo Zhu, Yuze Wu, Xijie Huang, Xiao Cui, Fei Gao, Xin Zhou
Abstract
Pretrained video foundation models encode rich semantic and spatiotemporal priors for embodied navigation, yet converting these priors into UAV motion typically requires expensive future-video synthesis and geometric reconstruction. We investigate whether the motion implicit in future visual prediction can instead be recovered directly from the predictive representations of a frozen video model. To this end, we present DiffWAM, a geometry-conditioned navigation world-action model that directly transforms multi-level predictive features into continuous camera trajectories. Its Grid-Motion module preserves spatial-temporal motion associations, while Latent2Pose grounds them with first-frame geometry to recover metrically meaningful 3D motion. Complete video rollouts and geometric reconstruction are required only for offline supervision, eliminating future-video decoding and multi-frame reconstruction during deployment. We further introduce FastDreamer, which overlaps predictive and geometric computation with ongoing flight and performs timestamp-aware asynchronous trajectory handoff for continuous UAV execution. DiffWAM achieves a trajectory RMSE of 0.3492 m and an endpoint success rate of 74.40% on the 1,000-sample DiffWAM-1000 benchmark, while representative real-world experiments demonstrate complex behaviors including constrained traversal, orbiting, S-shaped flight, and multi-stage navigation. An onboard DiffWAM-Flash implementation further reaches 1.08 s model-pipeline latency on NVIDIA Jetson AGX Thor. These results demonstrate that predictive video representations can be efficiently grounded into continuous 3D motion, providing a direct alternative to generate-then-reconstruct navigation pipelines. Project page: https://zzmmzzm.github.io/diffwam.github.io/.