Multi-modal video fusion improves quality and speed with dynamic regions
Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction
Computer Vision and Pattern Recognition
Summary
Combining videos from different sensors can create clearer and more consistent videos, but previous methods slowed down because they analyzed every part of the video each time. The authors designed a new method that focuses on parts of the video where things actually move, ignoring static areas to save time and boost quality. They also used advanced features from a pretrained model to reduce flickering between frames. Their approach works better and faster on popular test videos, making it suitable for use on smaller devices like phones.
What this means in practice
- •For mobile app developers: Create video apps that efficiently combine data from multiple sensors to deliver smooth, high-quality videos on resource-limited devices.$Commercial implications: Enables development of commercial multi-sensor video apps with superior fusion quality and lower computational cost suitable for mobile deployment.
- •For security system engineers: Improve real-time multi-camera video fusion for surveillance by focusing processing on areas with motion to enhance clarity and temporal consistency.
Authors
Jingxue Huang, Xiaosong Li, Yi Lin
Abstract
Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.