Rgb frame and event-voxel learning boosts video and text understanding
Leveraging Complementary Event Frame and Voxel Representations for RGB-Guided Multimodal Pre-training
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The paper addresses how to better combine traditional video frames with event-based data to understand motion and scenes over time. The authors propose a new method that teaches a computer to jointly learn from both types of data by focusing on important moving areas and reconstructing missing parts. They also use techniques that help the system generate video frames based on event data and create text descriptions from combined video and event information. This approach improves the computer's ability to analyze videos and link visuals with language.
What this means in practice
- •For video analytics developers: Develop video analysis tools that use both traditional frames and event data to improve motion understanding and scene recognition.
- •For multimodal ai engineers: Create systems that generate text descriptions from combined video and event data to enhance video indexing and search capabilities.
Authors
Wentao Wu, Xiao Wang, Yi Wang, Chenglong Li, Yonghong Tian, Bin Luo
Abstract
In this paper, we propose CM3AE 2.0, a unified RGB Frame and Event-Voxel/-Frame pre-training framework. By jointly learning multimodal spatiotemporal representations and exploiting latent cross-modal dependencies through cross-modal generation, the framework enhances multimodal information interaction and collaborative modeling capabilities. Specifically, we first propose an Event-aware Spatiotemporal Masked Reconstruction (ESMR) strategy to learn unified multimodal spatiotemporal representations. It leverages event responses to guide masking toward dynamically salient regions. Through spatiotemporal masked reconstruction, it learns spatial structures and their temporal variations from continuous RGB-Event observations. In addition, it incorporates multimodal fusion reconstruction and cross-modal contrastive learning to facilitate the modeling of cross-modal complementary information and the learning of shared representations. Building upon this, we further propose a Hierarchical Cross-modal Generation (HCG) strategy to learn latent cross-modal dependencies through visual and language generation. At the visual level, Event-conditioned RGB generation is performed using a pre-trained diffusion model to capture visual associations between the two modalities. At the language level, video-level text description generation based on joint RGB-Event representations establishes connections between multimodal visual features and linguistic semantics, further enriching the learning of cross-modal dependencies. Extensive experiments on four diverse downstream tasks demonstrate that CM3AE 2.0 consistently improves performance, validating the effectiveness and generalization capability of the proposed pre-training framework.