Foundation model separates learning for radar and optical images
SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models
Computer Vision and Pattern Recognition
Summary
Radar and optical cameras see the world in very different ways — radar works anytime and in any weather, while optical cameras capture fine details and colors. The paper presents a new method called SAREO-FM that treats radar and optical data separately but learns to combine their strengths. It uses separate pieces inside a model for understanding what is in the scene and for reconstructing the sensor signals, which helps the model learn better. This method improves how computers understand images from each sensor alone and works even better when combining both together.
What this means in practice
- •For remote sensing teams: Improve object detection and scene understanding by combining radar and optical images with separated semantic and reconstruction learning.
- •For autonomous vehicle developers: Enhance perception systems with day-and-night and weather-robust sensing by integrating radar and optical inputs using separate learning streams.
Authors
Jeonghyeok Do, Munchurl Kim
Abstract
Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves strong unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR--EO observations on tasks that benefit from complementary sensing.