Map-anchored prompting doubles drone success in object finding

Reaching Is Not Finding: Map-Anchored Visual Prompting for the Last Mile of Aerial Vision-Language Navigation

Computer Vision and Pattern Recognition

Summary

Drones often get close to a target but fail to find the exact object described in natural language, stopping near landmarks instead. The authors found this happens because current systems learn to reach landmarks rather than identify specific objects. They created a module called MAP-VP that draws landmarks on the drone's visual feed and uses a smaller vision-language model to pinpoint the exact target relative to these landmarks. This method significantly improves the drone's ability to find objects without retraining the original navigation systems.

What this means in practice

  • For drone navigation developers: Use MAP-VP to improve drones’ ability to precisely identify objects described relative to landmarks without retraining navigation models.
  • For autonomous robotics teams: Integrate map-anchored visual prompts for better last-step object localization in robots operating in landmark-rich urban environments.

Authors

Ningbo Li, Kebin Liu, Jingcan Chen, Weichen Zhang, Rong Ding, Yunhao Liu, Mo Li

Abstract

Aerial vision-language navigation (VLN) asks a drone to find an object described in natural language. The description usually locates the object relative to named landmarks, such as streets and buildings. We observe that current navigators often bring the drone close to the target and then stop at the wrong place: they reach the target without finding it. This gap is shared by navigators of very different designs, from imitation learners to a fine-tuned 7B vision-language model. A simple test reveals the cause. When we replace the target object in the instruction with another one next to the same landmarks, no navigator goes to it, and most stop where they did before. Navigators learn to reach the landmarks, not to identify the described object. We therefore propose MAP-VP, a plug-and-play module for the last step. After any navigator stops, MAP-VP draws the landmarks named in the instruction onto the drone's current view, using the landmark outlines that the benchmark provides on a map. A fine-tuned 4B vision-language model then locates the target relative to these landmarks. The training data are generated automatically from the benchmark annotations. Neither drawing the landmarks nor fine-tuning helps much alone, but together they are highly effective. Without retraining any navigator, MAP-VP significantly improves all six navigators we test on CityNav. It roughly doubles the success rate of the two strongest trained navigators, raising HETT from 25.9% to 54.6% and FlightGPT from 24.2% to 48.9%, whereas CityNav's official detector-based last step adds less than two points to HETT.