Physics based probing reveals what geo foundation models sense
PiCo: Physics-Informed Concept Probing for Geo-Foundation Models
Computer Vision and Pattern Recognition
Summary
Geo-foundation models analyze satellite data to predict things like forest types or ground cover, but it’s unclear which physical features actually influence their predictions. The authors propose a method called PiCo that uses simulated satellite images and physics-based models to find concepts the AI is sensitive to, like leaf water content or canopy structure. This helps explain how the models make decisions and why they succeed or fail on certain tasks. PiCo also shows how limiting input data, such as only using RGB images, affects the model’s ability to detect dry vegetation.
What this means in practice
- •For remote sensing engineers: Diagnose which physical features satellite image models rely on to improve model selection and tuning for environmental mapping.
- •For environmental monitoring teams: Evaluate whether model predictions for vegetation types remain reliable when constrained to simpler input data like RGB imagery.
Authors
Yihang She, Andrew Blake, Jianbo Qi, Srinivasan Keshav
Abstract
Geo-foundation models (GFMs) are typically evaluated by downstream task predictive performance, revealing little about which physical features they are sensitive to or use in prediction. This limits developers' ability to test hypotheses about model behaviour and users' ability to select models for new tasks. We propose Physics-Informed Concept Probing (PiCo), a post-hoc framework for explaining GFM predictions. PiCo identifies concept vectors in a GFM's embedding space by generating synthetic multispectral satellite observations from a 3D radiative transfer model coupled with a game engine. These vectors reveal physical sensitivities that inform domain-based hypotheses about downstream performance. Projecting embeddings onto these concept vectors produces low-dimensional representations for quantifying concept importance in making predictions. We evaluate PiCo on three GFMs with different input modalities using seven physical concepts covering forest leaf optics and canopy structure. In ground-cover estimation, RGB restriction reduces dry-matter sensitivity and disproportionately degrades predictions for dry vegetation and bare ground, supporting a hypothesis derived from sensitivity analysis. For hierarchical forest type and tree species classification, concept representations retain 70.0-93.8% of full-embedding accuracy. Models capture similar physical contrasts between classes, such as structural differences between broadleaf forest and cleared ground and leaf-water-content differences between oak and beech, yet differ in the strength of these contrasts and the concepts driving classification. PiCo thus provides a physical basis for hypothesis testing and model assessment, with potential applicability beyond Earth observation wherever forward models enable controlled simulation and pretrained embeddings are accessible for probing.