Audio-visual model improves egocentric action recognition despite sensor outages
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Computer Vision and Pattern Recognition
Summary
Recognizing actions from first-person videos often relies on both sound and visuals working together. But in reality, sensors can temporarily fail and come back on unpredictably, making it hard to rely on both streams all the time. The authors introduced MacJEPA, a model that learns to handle these temporary audio or visual dropouts during training. This helps it recognize actions and sounds accurately in long videos even when parts of the data are missing or noisy. The approach works well on egocentric video datasets without needing extra adjustment when deployed.
What this means in practice
- •For surveillance teams: Improve action recognition in bodycam videos despite intermittent sensor failures during long recordings.
- •For consumer electronics developers: Build wearable devices that reliably interpret user activities even when audio or visual sensors temporarily lose data.$Commercial implications: Enables development of robust activity recognition features in next-generation wearables by handling realistic sensor outages.
Authors
Souptik Sen, Zahra Ahmadi
Abstract
Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce \textbf{MacJEPA}, a missing-modality-robust \textbf{Ma}sked-\textbf{c}ontext query \textbf{JEPA} that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.