Relation constrained supervision improves multi modal image fusion quality
Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion
Computer Vision and Pattern Recognition
Summary
Multi-modal image fusion combines images from different sources into one, but it’s hard to know what the perfect combined image should look like because there’s no true reference. The authors found that common ways to teach fusion methods focus too much on matching pixel details, which can lead to bias or compromise between sources. Instead, they designed a way to teach fusion methods based on relationships between image features using pretrained models, making the teaching process better aligned with fusion goals. Their experiments show this new way helps fusion systems create better combined images regardless of the underlying network used.
What this means in practice
- •For medical imaging teams: Improve fusion of images from different medical scanners by providing better training signals that preserve important shared and unique details.
- •For remote sensing analysts: Enhance satellite image fusion by using relation-based supervision to reduce pixel-level compromises and modality bias in combined imagery.
Authors
Zeyu Wang, Mingyu Ge, Haiyu Song, Haoran Duan
Abstract
Multi-modal image fusion (MMIF) aims to form a single image by integrating shared information, preserving complementary cues, and coordinating cross-modal conflicts across modalities. However, due to the absence of ground-truth fused images, existing MMIF supervision commonly uses spatial-domain sources or gradient variants as surrogate ground truth, making the supervision mechanism inherently misaligned with the goal of MMIF and causing pixel-level compromise or modality bias. To address this, we propose a relation-constrained supervision paradigm that moves fusion supervision from the spatial domain to a learned relation space. Rather than relying solely on direct source approximation, we further leverage frozen pretrained representation models as information providers and design a learnable feature adapter to align heterogeneous DINO and CLIP features into a unified supervision space. The adapter infers three relation parameters, namely sharedness, dominance, and coordination radius, which define three losses corresponding to the MMIF's goal. To make this space reliable, we devise a self-supervised contrastive ranking objective tailored to the adapter and couple it with the fusion network through alternating optimization. Extensive experiments show that the proposed supervision space yields significant gains regardless of which mainstream backbone the fusion network adopts, offering a supervision paradigm better aligned with the goal of MMIF. Code: github.com/GMY628/RCS-Fusion.