Summary
Classifying scenes in satellite or aerial images using large multimodal language models can work well when the categories are known, but these models often make confident mistakes on new or unknown classes, a problem called silent hallucination. The authors designed a new method called RSAJEV that stores example images as prototypes and makes decisions by comparing new images to these stored examples without additional training. This approach greatly improves accuracy across different datasets and correctly identifies when an image doesn’t belong to any known class, which helps avoid mistaken confident predictions. RSAJEV also allows adding or removing classes easily without retraining the model.
What this means in practice
- •For remote sensing analysts: Improve classification accuracy and reduce false confidence when identifying land cover types in satellite images using existing language models without retraining.
- •For geospatial software developers: Implement a training-free prototype memory module to enable robust scene classification with out-of-vocabulary class handling in geospatial image analysis tools.
Authors
Man Chen, Antai Chen, Shuai Xu, Chengjie Gu, Huaping Zhou, Kelei Sun, Tao Wu, Yongjie Huang
Abstract
Scene classification of remote sensing imagery with multimodal large language models (MLLMs) is formulated as a single-forward-pass decision problem: candidate class names are lettered options in the prompt, a registered answer slot is conditioned on the prompt, and classification is read out by scoring letter tokens with the language-modeling head. We show that although this letter binding is highly effective within its training vocabulary (97.71% OA on UCM), it behaves erratically out of vocabulary: zero-adaptation accuracy across four cross-dataset directions fluctuates between 7.35% and 96.40%, and under out-of-vocabulary weights its confidence fails inversely on open-set evaluation, assigning higher confidence to unseen than to seen classes (AUROC 0.38), a failure we term silent hallucination. We retain the slot representation and replace the readout with RSAJEV, a training-free abstaining prototype memory that performs retrieval-based prototype matching: K reference images per class are cached as prototypes via their slot representations under the same prompt, and classification becomes nearest-neighbor retrieval over the prototypes, refined by a shrinkage-Mahalanobis geometric correction. This readout is mechanistically isomorphic to retrieval-augmented generation (RAG) but operates on discriminative decisions; it is the first retrieval-based readout for MLLM slot representations. The retrieval readout reaches 89.43% on the UCM cross-dataset gate with 16 references per class and zero gradient updates, and 96.90% on AID. Its ID-ness scores attain 0.91-0.99 AUROC on open-set evaluation over two datasets, exceeding symbolic confidence (0.38-0.96) in all four open-set settings, restoring a calibratable abstention channel and supporting class addition/removal at zero extra inference cost via offline rescoring of the cached matrix.