SeeRF fuses images beyond sensor field-of-view limits
SeeRF: Learning to See Beyond Fields of View via Generative Image Registration and Fusion
Computer Vision and Pattern Recognition
Summary
Many cameras and sensors can only capture parts of a scene, making it hard to combine their images fully without missing or mismatched areas. The authors present SeeRF, a new method that smartly aligns and merges images from different sensors, even when their views don't overlap. It uses advanced models to fill in missing parts and create natural, complete scenes. This helps combine information better than before when images only partly cover the same area.
What this means in practice
- •For autonomous vehicle teams: Integrate images from different sensors with non-overlapping views for safer and more complete environment perception.
- •For security system developers: Create fused surveillance images that cover beyond individual camera limits, improving situational awareness in monitoring.
Authors
Xunpeng Yi, Yibing Zhang, Qinglong Yan, Yifan Lu, Han Xu, Jiayi Ma
Abstract
Due to the inherent discrepancies in imaging viewpoints and field-of-view mismatches among multi-modal sensors, existing fusion methods have long struggled to achieve complete scene-level information integration beyond the limitation in sensor fields of view. Consequently, the generated fusion results often contain modality-specific regions and fusion artifacts, limiting the comprehensive representation of scene structures and semantic information. To address this challenge, we propose a generative registration-fusion framework, termed SeeRF. Specifically, it first introduces a deformation generation strategy based on the diffeomorphic stationary velocity field, combined with visual foundation model-guided feature extraction and a coarse-to-fine progressive multi-modal image registration module, to achieve accurate cross-modal flow estimation and generate valid field-of-view masks. Subsequently, the registration module is tightly coupled with the generative diffusion module for multi-modal completion and fusion, where the cross-modal control encoding module and the control residual feature encoder-decoder mechanism provide effective guidance. By leveraging the powerful prior modeling capability of generative diffusion models, the proposed framework overcomes the limitations imposed by shared fields of view and enables effective completion and natural fusion of non-overlapping field-of-view regions. Extensive experiments on multiple public datasets and real-world scenarios demonstrate that the proposed method achieves superior performance, validating its effectiveness, superiority, and robustness. The code will be available at https://github.com/XunpengYi/SeeRF.