UAV navigation improved by imagining next visual scenes from language
ProtoVis-Nav: Prototypical Visual Imagination for Visual-Spatial Aligned UAV Navigation
RoboticsComputer Vision and Pattern Recognition
Summary
Flying drones using instructions is hard because small mistakes in guessing where to go next can cause crashes. The authors created a method that helps drones imagine what the next place should look like, based on the instructions they get. This helps the drones better understand where to move by predicting visual features before actually moving there. Their tests show that this way of combining language instructions with imagined visuals improves navigation accuracy.
What this means in practice
- •For drone software developers: Enable drones to predict and visualize the next movement location to navigate more accurately from language commands.
- •For robotics system integrators: Improve UAV navigation systems by integrating local visual imagination models for safer autonomous flight.
Authors
Guangyuan Sui, Yunzhong Hou, Chi Harold Liu
Abstract
Vision language navigation (VLN) is very challenging for Unmanned Aerial Vehicles (UAVs) since small errors in 3D motion prediction can lead to navigation errors and collisions. Unable to fathom execution outcomes in 3D environments, existing methods often struggle to predict accurate navigation waypoints from language instructions. To solve this, we introduce the Prototypical Visual Imagination framework, shortened as ProtoVis-Nav, which forces the model to imagine an outcome-associated local visual representation for the intended next UAV movement. Specifically, we predict feature prototypes and use their linear combination to estimate the visual feature of the target region. In this manner, the VLN model first imagines the local visual representation of the next intended location, and then uses predicted prototypes as context for generating the final action, promoting alignment between visual context and waypoint outputs. Extensive experiments demonstrate the effectiveness of the ProtoVis-Nav framework and show substantial improvements over existing baselines.