Multimodal image fusion improves by separating detail and structure features
RoES: Rotational Equivariant Selective-frequency Fusion for Multimodal Images
Computer Vision and Pattern Recognition
Summary
Combining infrared and visible images helps machines see better by mixing heat patterns with regular pictures. Existing methods treat all image features the same when rotating, but this can blur unique fine details. The paper introduces RoES, a new way to smartly split image parts by their detail level and handle rotation differently for each. This leads to clearer combined images and helps with tasks like detecting objects more accurately.
What this means in practice
- •For surveillance system engineers: Create fused images from visible and infrared sensors that maintain structural consistency and fine details under rotation for better object tracking.
- •For autonomous vehicle developers: Improve perception by generating robust multimodal images that enhance object detection under diverse viewing angles and lighting conditions.
Authors
Jiabao Wang, Wenjian Liu, Yaoming Cai, Gengyu Zhang, Boyan Zhao, Zijia Zhang, Yao Ding, Xiaobo Liu
Abstract
Infrared-visible image fusion facilitates robust multimodal perception by integrating complementary textural nuances from visible sensors with thermal signatures from infrared systems. Due to the task's inherently ill-posed nature, existing methods heavily rely on structural priors but typically enforce rotation equivariance uniformly across all features. Such a holistic approach overlooks a critical distinction where low-frequency shared structures strictly adhere to equivariant constraints while high-frequency modality-specific details require greater flexibility to preserve unique information. To bridge this gap, we propose RoES, a Rotational Equivariant Selective-frequency fusion network. Instead of employing static decomposition, we introduce a trainable rotation-enhanced updater/predictor module to dynamically decouple low- and high-frequency components. The resulting representations are then processed through a dual-branch fusion module tailored for spectral consistency. Specifically, a rotation-equivariant Mamba is employed to capture long-range structural dependencies in the low-frequency domain, while a polar spectral attention-based Dual-Fourier block refines high-frequency details under explicit low-frequency guidance. Extensive experiments demonstrate that RoES consistently achieves state-of-the-art performance in both fusion quality and downstream object detection, establishing a robust solution for multimodal fusion by reconciling frequency-selective features with equivariant constraints. The source code is available at https://github.com/BryceLosky/RoES-Fusion.