Feature distillation improves geospatial models for multiple sensors
Reuse, Don't Retrain: Efficient, Modality-Versatile Geospatial Foundation Models through Feature Distillation
Computer Vision and Pattern Recognition
Summary
Training large geospatial AI models from scratch is expensive and often repeated for different image types and sizes. The authors show it is more efficient to reuse existing large models by teaching smaller ones to copy their features, a process called feature distillation. These smaller models match or outperform bigger original models but require far less computing power to train. They also combine knowledge from different models that handle multispectral and RGB images into one versatile model without needing aligned data. This makes geospatial AI models more accessible and capable across diverse sensing types.
What this means in practice
- •For remote sensing developers: Create efficient geospatial AI models that perform well across diverse sensor types with less computing cost by leveraging feature distillation.
- •For agriculture monitoring teams: Improve crop and land analysis accuracy using a single AI model that integrates multispectral and high-resolution RGB satellite data without needing aligned images.
Authors
Mario Hamon, Pierre Adorni, Minh-Tan Pham
Abstract
Geospatial foundation models (GeoFMs) are costly to pretrain, yet model families often repeat this process across backbone scales and sensing modalities. We investigate feature distillation as a more efficient way to reuse existing representations. First, we distill frozen CROMA and TerraMind teachers into smaller Vision Transformers through patch-level feature matching. Across nine PANGAEA tasks, distilled TerraMind Tiny, Small, and Base outperform their scale-matched released counterparts by 2.76, 3.02, and 1.87 mean IoU points, respectively; each also matches or exceeds the next-larger released model. Distilled CROMA-Base gains 2.05 points. TerraMind-Base required only 43 A6000 GPU-hours, approximately 0.5\% of the reported cost of independent pretraining. Second, we introduce TEDI, an unpaired multi-expert approach combining TerraMind for multispectral imagery and DINOv3 for high-resolution RGB. Separate patch embeddings route unpaired inputs into a shared Transformer trunk, removing the need for co-registered data. TEDI-Large achieves the highest single-model mean in our PANGAEA-9 comparison (62.51 mIoU), outperforming both teachers overall while retaining their modality-specific strengths. TEDI-Base reaches 61.75 mIoU with 72\% fewer parameters. These results show that distillation improves the GeoFM accuracy-compute frontier while consolidating complementary experts into a single versatile backbone.