AI summaryⓘ
The authors studied how well geospatial AI models (GeoFMs) perform not just on average accuracy but also on how well their confidence matches their actual correctness, especially when data quality worsens. They found that all models get worse with data corruption, and models pretrained on Earth observation (EO) data are not more stable than those pretrained on ImageNet. EO models tend to become too confident when their accuracy drops, linked to their less flexible internal representations. Common methods to fix confidence issues like temperature scaling or ensembles don't fully solve the problem. The authors suggest evaluating models using many metrics and conditions to better reflect real-world uses.
Geospatial Foundation ModelsEarth ObservationModel CalibrationData CorruptionConfidence EstimationCentered Kernel AlignmentTemperature ScalingDeep EnsemblesSelective PredictionExpected Calibration Error
Authors
Nils Lehmann, Jakob Gawlikowski, Burak Ekim, Isaac Corley, Xiao Xiang Zhu
Abstract
Geospatial Foundation Models (GeoFMs) are most commonly ranked and selected by accuracy on standard benchmark conditions via averaged ranks. We show that this protocol is too narrow: the promised deployment in critical EO tasks requires further angles of analysis, mainly calibration, the agreement between a model's confidence and its correctness. Across 16 frozen encoders, four classification and five segmentation datasets, and two orthogonal stress axes, every encoder degrades as corruption intensifies, and the ranking changes as well. Across the four classification benchmarks, EO-pretrained and ImageNet-pretrained encoders are indistinguishable on clean accuracy and clean calibration, and EO pretraining provides no more stability under shift than ImageNet pretraining. Under shift the GeoFMs drift further into overconfidence than the ImageNet-pretrained encoders, at every grade and in every corruption family. A centered kernel alignment (CKA) analysis ties this to representational rigidity: EO-pretrained embeddings move less under corruption while losing just as much task information and remaining overconfident. We apply three commonly explored uncertainty quantification methods and find that temperature scaling and deep ensembles cannot counteract the degradation, while a Gaussian-process probe roughly halves ECE under severe cloud only by tripling it on clean data. In selective prediction experiments, we find that confidence-based abstention cannot defer around confidently wrong predictions, and advocate that benchmark rankings and evaluations should therefore operate across a multitude of conditions and metrics to more holistically evaluate model development progress and close the gap to real world deployment scenarios.