Foundation models adapt better to remote sensing by learning flexible subspaces
Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensing
Computer Vision and Pattern Recognition
Summary
Foundation models are big AI systems pre-trained on lots of data, but they can be expensive to customize for new tasks. This paper finds that typical methods for fine-tuning them don't work well when remote sensing images are very different from the original training data. The authors propose a new way to adapt these models by learning flexible coordinate systems that better match the new data's patterns, making fine-tuning more effective. Their approach also works when combining different types of data, like images and other sensors, and it performs better than traditional fine-tuning without slowing down the model.
What this means in practice
- •For remote sensing engineers: Adapt pretrained vision models efficiently to new remote sensing tasks with domain shifts, improving accuracy without added inference cost.
- •For multimodal ai developers: Develop systems that combine data from different sensor types more effectively by using shared adaptive subspace transformations.
Authors
Han Luo, Ruoyu Yang, Yinhe Liu, Yanfei Zhong
Abstract
Pretrained foundation models (FMs) have achieved remarkable success in computer vision, yet their high fine-tuning cost limits practical deployment. Parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) improve efficiency by constraining updates to a predefined low-rank subspace. However, when applied to remote sensing tasks with substantial domain shifts, the fixed subspace is constructed without observing the downstream activation distribution and can therefore provide a poor coordinate system for adaptation, a phenomenon herein termed subspace mismatch. To address this issue, a unified framework is introduced, termed Domain-aware Relaxed Orthogonal Subspace adaptation (DROS), which reformulates low-rank adaptation as data-conditioned subspace learning and flexible subspace adaptation. Specifically, the weight decomposition is conditioned on second-order activation statistics estimated from the downstream training distribution, so that the initialization reflects the feature geometry actually induced by the remote-sensing data, followed by flexible geometric transformations enabled by a relaxed orthogonal parameterization. Furthermore, the framework is extended to multimodal settings (MM-DROS) by sharing transformation structures across modality-specific subspaces, facilitating efficient cross-modal interaction. Extensive experiments on multiple remote sensing benchmarks demonstrate that DROS achieves state-of-the-art performance, even surpassing full fine-tuning, without additional inference overhead.