Multimodal image fusion improved by signal level feature decomposition
When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion
Computer Vision and Pattern Recognition
Summary
Combining images from different sources, like infrared and visible light, can create clearer pictures useful for many tasks. The researchers found that existing methods to mix these images often struggle because they don’t have clear examples to guide them. They created a new way to break down image features into common and unique parts using one-dimensional signals instead of full images, making it easier for the system to learn. Their approach includes two learning stages that help produce better fused images, showing strong performance in tests.
What this means in practice
- •For medical imaging teams: Improve clarity in combined medical images from different scanners by better separating shared and unique image details without needing ground-truth data.
- •For surveillance system developers: Create fused security footage integrating infrared and visible cameras with enhanced detail preservation enabled by signal-level feature decomposition.
Authors
Zeyu Wang, Jiayu Wang, Haiyu Song, Haoran Duan
Abstract
Multimodal image fusion (MMIF) aims to integrate complementary information from different modalities into a high-quality fused image and support downstream tasks. Recently, feature decomposition has become an important paradigm by separating source images into common and modality-specific unique features. However, existing methods lack clear supervision because ground-truth (GT) decomposition feature maps are unavailable. They usually combine multiple image-level metrics as losses, which are inherently incomplete and may conflict since each pixel couples attributes such as texture, edge, and contour. To address this, we propose a 1D signal-level self-supervised feature decomposition paradigm. Our core insight is to reformulate feature decomposition from unclear 2D image-level supervision into an integral-driven 1D signal-level optimization problem. This objective-level reformulation uses the 1D signal form to compute the integral constraint. The decomposer is optimized by the integral area between common and original signals, enabling more stable optimization with a clear optimization objective. Our model follows a two-stage SSL framework. Stage I designs dual pretext tasks for integral-driven decomposition at the signal level and structure-preserving reconstruction at the image level. Stage II fuses unique features and combines them with common features to reconstruct the fused image. Experiments on representative MMIF tasks show state-of-the-art (SOTA) performance. Code: github.com/Wangjiayu0512/SIDFusion.