Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors
2026-06-22 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors study how machine learning models used for satellite data can adapt when the types of sensors (modalities) change, like when new sensors are added or old ones removed. They identify three main cases: swapping sensors, adding more sensors, or using fewer sensors. To handle these changes, they present DeluluNet, a flexible model that can learn to guess missing sensor data from the ones available without needing extra labeled data. This way, the model keeps working well even when the input sensors change, avoiding the need for time-consuming retraining.
machine learningremote sensingmodalitiesmodality transfermodality additionmodality peekingmultimodal learningmodality hallucinationend-to-end trainingsatellite sensors
Authors
Tim G. Zhou, Anthony Fuller, Geoff Pleiss, Evan Shelhamer
Abstract
Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination--predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.