Dynamic RGB thermal reconstruction tracks heat and color changes in 3D scenes
Dynamic Thermal Gaussians: Multimodal 4D Gaussian Splatting
Computer Vision and Pattern Recognition
Summary
Thermal imaging helps us see heat patterns, but most tools only look at still scenes and miss how temperature changes over time. The authors created a new method that combines normal color images and thermal data to show how both appearance and heat move in 3D spaces. Their approach uses a shared 3D model to keep color and heat aligned even as the scene changes. They also made a special dataset to test their method, showing it can accurately capture detailed temperature changes along with what the scene looks like.
What this means in practice
- •For thermal monitoring teams: Produce accurate 4D visualizations of heat variations in dynamic environments combining color and thermal data for better analysis.
- •For security system integrators: Create thermal-aware RGB security cameras that detect moving heat sources with improved spatial and temporal detail.$Commercial implications: Enables development of advanced surveillance products integrating synchronized thermal and RGB imaging for real-time heat detection.
Authors
Rongfeng Lu, Lifeng Lin, Xiaobao Wei, Quan Chen, Ming Lu, Yitian Xue, Yaoqi Sun, Yuhan Gao, Anke Xue, Chenggang Yan
Abstract
Thermography plays a vital role in military and broader thermal analysis applications. Recent progress in 3D thermal reconstruction has extended temperature analysis from 2D to 3D space, yet most existing works assume static temperature distributions, neglecting the temporal dynamics of heat transfer in real-world environments. To address this limitation, we propose the first dynamic RGB-Thermal reconstruction framework for complex scenes. Our method jointly models RGB appearance, thermal observations, and scene geometry as they change over time. Specifically, we introduce a multimodal dynamic scene representation that anchors both the color and thermal modalities to a shared geometric substrate, ensuring their consistency under spatiotemporal deformations. We further design multimodal embeddings to enhance the motion expressiveness for each modality, and propose a multimodal routing mechanism that retains a unified set of shared multimodal Gaussians as the geometric backbone while adaptively spawning modality-specific Gaussians to strengthen the representational capacity in detail-rich regions of each individual modality. In addition, we contribute a novel benchmark dataset featuring high-frequency temperature variations to facilitate the evaluation of 4D reconstruction. Extensive experiments demonstrate that our method achieves high-fidelity spatiotemporal reconstruction of both appearance and temperature. Our code and dataset are available at: https://github.com/LinLif1869/DTG.