Multi-modal image registration improved with shared structural feature learning

RegNC-Mamba3: Shared Structural Feature Learning Using Non-Causal State Space Principles for Multi-Modal Image Registration

Computer Vision and Pattern Recognition

Summary

Matching images from different sources, like visible light and infrared, is hard because their features look different. The authors created a new method that better picks out common shapes and structures in these images using a special mathematical approach called non-causal state space principles. This method looks at image data from many angles and frequencies to find shared details more accurately and quickly. Tests showed their approach registers images better than other methods.

What this means in practice

  • For medical image analysts: Improve alignment of different medical imaging types like MRI and CT scans for better diagnosis by more accurately detecting shared structures.
  • For remote sensing engineers: Enhance alignment of multi-source satellite images such as visible and radar for improved environmental monitoring and mapping.

Authors

Zhikang Li, Yan Wu, Xin Hu, Yi Dai, Ming Li

Abstract

The main challenge of multi-modal image registration is shared structural feature extraction with higher accuracy and lower computational complexity. Compared with a State Space Model (SSM), the State Space Principles (SSP) formulation of Mamba-3 uses a more accurate discretization with lower truncation error while maintaining linear complexity. However, vanilla SSP is designed for natural language processing (NLP) and is not directly suitable for images. To address this issue, we propose a novel non-causal SSP (NC-SSP) with shared structure awareness and develop a coarse-to-fine registration method named RegNC-Mamba3. First, NC-SSP is designed to eliminate the dependence of the integral computation on the scanning path. By combining relative-weight normalization with aggregation over all circular scan paths, NC-SSP constructs a globally hidden state without long-range decay, which provides a wider receptive field and higher efficiency than the original Mamba-3. Second, NC-SSP is modified to strengthen the extraction of cross-modal structural features. When computing hidden states and feature interactions, we construct complex-valued sequences in NC-SSP by applying learnable Log-Gabor kernels, with different orientations to each head for frequency-domain modulation. This improves the perception of shared structural information across modalities and increases both registration accuracy and quality. Experiments on VIS-SAR and VIS-IR demonstrate that RegNC-Mamba3 outperforms the compared detector-based and semi-dense multi-modal registration methods.