Models often provide unsupported answers in remote sensing questions
Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering
Computer Vision and Pattern Recognition
Summary
Some computer programs try to answer questions about satellite images and also show which parts of the image support their answers. The researchers found that even when these programs give the right answer, the parts they point to often do not actually support that answer. They created a new test to check if the answers really match the evidence the program points to. They also built a large set of example questions and images to measure this problem.
What this means in practice
- •For satellite image analysts: Improve trustworthiness of AI answers by verifying that identified image evidence supports answers to remote sensing queries.
- •For geospatial ai developers: Create more reliable AI systems for remote sensing that align their textual answers with correct visual evidence in images.
Authors
Ping Guo, Zhiqi Huang, Xinran Li, Linkun Yu, Jingyin Pei, Kamata Seiichiro
Abstract
In grounded remote sensing visual question answering, a model produces a textual answer and identifies relevant objects in the image as grounding evidence. Existing evaluations measure answer correctness and localization quality, but do not directly assess answer-grounding consistency. We propose the Answer-Evidence Support Protocol (AESP) to assess this consistency through evidence support: whether the grounding evidence supports the model's own answer. AESP applies task-specific rules to existence, counting, and comparison questions. We further construct the Grounded Remote Sensing Visual Question Answering (G-RSVQA) benchmark, containing 15,027 samples with questions, reference answers, and all related object-level annotations. Experiments on seven grounded vision--language models reveal failures of evidence support across all three question types. Up to 50.82% of correct answers in the evidence evaluation subset are not supported by the provided grounding evidence.