MemoryFusion improves video fusion with long term memory and smoothing
MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion
Computer Vision and Pattern Recognition
Summary
Multimodal videos combine information from different sources but existing methods focus only on short-term frame interactions, missing important long-term details. The authors propose MemoryFusion, a two-part system that remembers both recent and distant frames to create better combined video content. Their method also smooths videos over time to reduce flickering and improve stability. Tests show MemoryFusion produces clearer images and more consistent videos than prior methods.
What this means in practice
- •For video editors: Enhance video editing tools by integrating MemoryFusion to improve the quality and stability of fused multimodal footage over long time spans.$Commercial implications: Enables creation of products offering advanced video fusion and stabilization features for professional video production.
- •For autonomous vehicle developers: Use MemoryFusion to fuse multimodal video inputs with long-term context, reducing temporal inconsistencies that can affect perception systems.
Authors
Meiqi Gong, Hao Zhang, Jiayi Ma
Abstract
Multimodal videos contain rich temporal cues that can facilitate faithful fusion. However, existing video fusion methods mainly rely on short-range temporal interactions, leaving valuable long-term information underexplored. Moreover, inconsistent fusion representations can cause flickering artifacts, undermining temporal coherence. To address these coupled challenges, we propose MemoryFusion, a two-stage framework integrating cross-temporal memory learning with progressive temporal refinement. Considering the complementary roles of adjacent and distant frames, the first stage combines a Short-Term Memory Bank for local temporal guidance with a Long-Term Memory Bank that preserves reliable historical information through a fusion-oriented update strategy. To mitigate temporal fluctuations, a Temporal Smoothing Module leverages soft motion cues for coarse background stabilization. Building upon the resulting fusion prior, The second stage further employs a lightweight 3D residual Temporal Refinement Module to correct remaining temporal inconsistencies while enhancing structural details. Extensive experiments demonstrate that MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based fusion methods. The source code is publicly available at https://github.com/Meiqi-Gong/MemoryFusion.