Task specific vision language models generate paths on remote sensing images

RemoteVLPG: How to Obtain a Specific VLM for Language-Guided Path Generation on Remote Sensing Images?

Computer Vision and Pattern Recognition

Summary

Generating paths between two points on aerial or satellite images is challenging because it needs understanding both images and language instructions. The authors studied how to create specialized vision-language models that follow language instructions to draw paths on these images. They built a diverse dataset and tested different model designs, showing their approach works well even without extra training on specific tasks. Their models can outperform some commercial systems in zero-shot scenarios, meaning they work without specialized retraining.

What this means in practice

  • For geospatial analysts: Automatically generate paths on satellite images guided by language, improving remote routing and planning accuracy.
  • For mapping software developers: Integrate language-based path generation in mapping tools to allow users to describe a route and get it drawn on aerial imagery.

Authors

Junzhe Dang, Mengmeng Zhang, Yuxiang Zhang, Wei Li

Abstract

We study how to obtain a task-specific Vision-Language Model (VLM) for language-guided path generation between start and end points on remote sensing images. We explore this question from three aspects. First, we implement three widely used architectural approaches and compare their performance. Second, we design a scalable method for constructing pre-training data and build an effective pre-training dataset with a degree of diversity. Methods trained on this dataset achieve promising zero-shot performance. Third, we construct three benchmarks that support stratified evaluation and design three metrics for path generation. We investigate this problem through extensive experiments. The results show that the zero-shot performance of methods trained on the constructed pre-training dataset can exceed that of some advanced commercial VLMs. Code and Dataset can be available at https://github.com/jzdang/RemoteVLPG