Unified model improves captioning question answering and grounding for satellite images
GeoNLI - A Natural Language Interpreter for Satellite Imagery
Computer Vision and Pattern Recognition
Summary
Understanding satellite images requires different tasks like describing the image, answering questions about it, and pointing out locations within it. The authors tested various models and combined them into one system that does all these tasks together. This combined system answers questions and captions images accurately, and it finds specific spots in images better than earlier separate methods. By mixing different AI models and making them vote on answers, the system becomes more reliable at interpreting satellite pictures.
What this means in practice
- •For geospatial analysts: Provide a single tool that accurately captions, answers questions on, and grounds objects in satellite images.
- •For climate monitoring teams: Improve automated interpretation of satellite images for tracking environmental changes and events.
Authors
Ashutosh Gandhe, Anupam Rawat, Geet Sethi, Kabir Nasiruddin, Madhav Kotecha, Panav Shah, Rakshit Sawarn, Soumitra Nayak
Abstract
Multi-modal multitasking models have shown strong performance on remote sensing datasets. However, because these models are trained on heterogeneous data and vary across tasks, designing a unified model that performs well in captioning, visual question answering (VQA), and visual grounding remains challenging. In this work, we evaluate several models on the VRS Bench and NWPU-VHR-10 datasets. The EarthMind model demonstrates strong results in both captioning and VQA. For grounding, we propose multiple pipelines - RemoteSAM-SAM-v1, RemoteSAM-SAM-v2, and DiffuSAM - and ultimately adopt a majority-voting ensemble across EarthMind, RemoteSAM, SAM3, Falcon, RemoteSAM-SAM3-v1, RemoteSAM-SAM3-v2, and DiffuSAM predictions. Our unified, modular pipeline integrates advanced SAM variants with multimodal LLMs to jointly perform captioning, VQA, and grounding. It achieves 82% accuracy on captioning and 83.32% on VQA, with 90.94%, 52.04%, and 92.06% for binary, numeric, and semantic question types respectively. For grounding, it attains 64.94% accuracy. By combining diverse VLMs with our custom RemoteSAM-SAM3 models through ensemble majority voting, the system delivers more accurate and consistent remote-sensing understanding than task-specific approaches.