AI summaryⓘ
The authors focus on improving how computers learn from videos that have multiple types of data, like regular video, infrared, depth images, and skeleton tracking, without needing labeled examples. They noticed that usual methods mistakenly treat all mismatches as equally wrong and all matches as perfect, which isn’t true when cameras change angle or data is noisy. To fix this, the authors made a system that gently scores how good or bad each match is, using clever consistency checks, and combines information from all data types together. Their method works better than older ones on a driving activity dataset and handles new camera views well, making it useful for detecting driver distraction reliably.
self-supervised learningmulti-modal datacontrastive learningInfoNCEcycle-consistencyglobal alignmentdriver distraction detectionDrive&Act datasetRGBcross-view generalization
Authors
David J. Lerch, Livien Majer, Zeyun Zhong, Manuel Martin, Frederik Diederichs, Rainer Stiefelhagen
Abstract
Robust self-supervised learning of multi-modal video representations is critical for real-world applications such as driver distraction detection, where multiple sensors provide complementary but noisy signals. Conventional contrastive objectives, such as InfoNCE, assume all negatives are equally informative and all positives are reliable. However, this assumption is frequently violated in multi-modal data due to viewpoint changes, occlusions, or semantic overlap across modalities. In this work, we propose a novel framework for multi-modal global alignment that addresses these challenges by jointly modeling faulty negatives and unreliable or faulty positives. We introduce soft targets derived from cycle-consistency scores to relax the hard-negative assumption, and a weighting mechanism based on similarity distributions to mitigate the impact of noisy or faulty positives. Our approach extends traditional pairwise alignment to a principled global multi-modal setting, aggregating alignment information across all modality pairs. We evaluate our method on the Drive&Act dataset, demonstrating that it consistently outperforms both pairwise and existing global alignment baselines across RGB, IR, Depth, and Skeleton modalities. Cross-view ablation studies further show strong generalization to unseen camera perspectives, highlighting the robustness of our representations. Overall, our framework provides a scalable and effective solution for self-supervised global multi-modal representation learning, enabling reliable driver distraction detection and pioneering in real-world multi-modal video understanding. Our code will be published on GitHub.