Multimodal fusion improves breathing and pose detection accuracy
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
This paper looks at how to combine signals from different sensors—video, depth, and radar—to better recognize breathing and body poses. The researchers used a shared model that can handle missing sensors, training it to perform well even when some data is unavailable. They found that training on all sensor subsets together boosts accuracy, while some advanced fusion techniques didn’t show clear improvements. Their method works well across multiple people and sensor combinations.
What this means in practice
- •For health monitoring developers: Build systems that accurately detect breathing patterns and body poses using combined video, depth, and radar sensors even when some sensors fail.
- •For surveillance system engineers: Create multimodal surveillance solutions that maintain recognition performance when certain sensor types are missing or temporarily unavailable.
Authors
Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen, Manuel Lage Cañellas, Miguel Bordallo López
Abstract
Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.