UAV navigation aligns language commands with diverse safe flight paths
VLM-MPPI: Grounding Natural Language in Behaviorally Diverse Trajectories for Aerial Navigation
Robotics
Summary
Flying drones indoors is tricky because they need to avoid obstacles while following instructions. The authors created a system that plans six different safe flight paths based on what the drone is told to do. They turn the problem of understanding language into picking the best flight path by matching pictures the drone sees with the instruction using a visual-language AI model. Their system works well both in computer simulation and on a real drone, flying safely and successfully every time.
What this means in practice
- •For drone developers: Provide drones the ability to interpret spoken or typed commands into diverse safe flight behaviors in indoor environments.
- •For robotics integration teams: Combine vision-language models with motion planning to improve task success and safety in autonomous UAV operations.
Authors
Hanbing Zhang, Fangguo Zhao, Zerui Li, Xin Guan, Peng Cheng, Shuo Li
Abstract
We present a hierarchical UAV navigation framework that aligns natural-language intent with dynamically feasible flight behaviors in cluttered indoor environments. To bridge the gap between abstract semantics and low-level control, we employ a parallelized ensemble of six behavior-conditioned Model Predictive Path Integral (MPPI) planners. Crucially, by designing mode-specific guiding costs and sampling biases, we induce distinct trajectory modes that converge to unique behavioral means, yielding a compact set of intentionally diverse candidates rather than mere stochastic variations. We project these 3D candidates onto the onboard first-person-view RGB stream, turning language grounding into a visual action selection problem. A pretrained vision--language model (VLM) asynchronously selects the candidate index given the overlaid FPV image and a natural-language prompt, while MPPI replans at 20Hz and a PID-based low-level controller tracks the selected trajectory. We implement the full pipeline in NVIDIA Isaac Sim and on a real-world quadrotor platform equipped with LiDAR and RGB sensing. Experiments in both simulation and real-world flights show semantically meaningful behavior diversity, robust language alignment despite VLM latency, and safe, repeatable flight across all modes, achieving 100% task success in our evaluated scenarios.