XCalib enhances alignment of infrared and visible camera videos
XCalib Depth-Guided Geometric Optimization for Dense Thermal-Visible Video Registration
Computer Vision and Pattern Recognition
Summary
Aligning infrared and visible images is important for things like driver-assistance systems but is tricky due to differences in camera position and angle. The authors propose XCalib, a new method that uses camera setup and depth information to better match these images over time. Unlike past approaches, XCalib constrains how the images can be warped to physically plausible ways, improving stability and accuracy. They also introduce a new way to measure how well the images line up, especially across different types of cameras.
What this means in practice
- •For automotive engineering teams: Improve thermal-visible camera alignment in driver-assistance systems for more accurate fused sensor data during dynamic driving conditions.
- •For security system integrators: Enhance the registration of infrared and RGB video streams in surveillance setups to reduce misalignment artifacts and improve object tracking accuracy.
Authors
Aurelien Godet, Gabriel Jobert, Mauro Dalla Mura
Abstract
Image registration is a vital preprocessing step in multimodal perception tasks, including image fusion, object detection, and semantic segmentation. In Advanced Driver- Assistance Systems (ADAS), spatial misalignment between visible (RGB) and infrared (IR) cameras -caused by non-coincident optical axes and field-of-view differences- introduces non-uniform parallax and visual ghosting. Classical keypoint-based methods are restricted to global homographies that fail under dynamic depth, while unconstrained dense flow algorithms lack structural regularization and suffer from temporal instability. In this paper, we propose XCalib, an unsupervised dense thermal-visible registration framework that bridges this gap. Rather than serving as an absolute metric calibration tool, XCalib leverages virtual pinhole camera parameterization strictly as a geometric constraint space. By optimizing effective relative pose and intrinsics alongside predicted monocular metric depth, XCalib restricts the search space of spatial displacements to physically valid projection geometries. Our key contributions are: (1) a novel registration paradigm that uses camera parameterization as an implicit regularizer for dense cross-modal warping; (2) Normalized Edges Correlation (NEC), a robust structural similarity metric tailored to cross- spectral alignment; and (3) extensive quantitative and qualitative evaluations across public ADAS datasets, demonstrating superior temporal stability and alignment accuracy over unconstrained dense flow baselines.