Egocentric video embeddings reveal head orientation and improve activity recognition
What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition
Computer Vision and Pattern Recognition
Summary
This paper looks at what video features from head-worn cameras really tell us, especially when combined with body sensor data like limb movements. The authors found that video features mainly show where the person is looking, while body sensors better capture the actual movements of limbs. By cleverly connecting video data and sensor data in a special graph and spreading predictions across it, they improved the accuracy of recognizing human activities. Mixing the data at the classifier level didn’t work as well as their graph-based approach.
What this means in practice
- •For wearable device developers: Improve human activity recognition by combining head-camera video features with body-worn inertial sensor data using graph propagation techniques.$Commercial implications: Enables more accurate activity tracking in wearables for fitness and health products by integrating video and sensor data.
- •For sports analytics teams: Identify and segment distinct exercise sets within athletes using combined video and inertial sensor data processed with a video graph.
Authors
Kansuke Habano
Abstract
Egocentric video and body-worn inertial sensors are increasingly recorded together, and the video is usually encoded by pretrained models. What these embeddings represent when the camera sits on the head, and how they should be combined with inertial sensing, has received little attention. Across VideoMAEv2, CLIP, DINOv2 and the egocentric VC-1, the leading principal directions of the embedding are dominated by where the camera looks. Measured head orientation explains them on Ego-Exo4D for egocentric (R^2 0.49-0.70) but not exocentric cameras, and the scene layout explains them on WEAR. The part that the limbs' inertial signals predict separates activities across people. In a set-level probe, the rest separates exercise sets within a participant but not activities across people. This is consistent with separate roles: the video finds the clips of the same set, and the inertial classifier labels them. On WEAR, with temporal order removed and one random limb per one-second clip, we propagate inertial predictions over a within-participant video graph whose leading components are removed. This raises subject-wise cross-validated macro-F1 from 0.648 to 0.852. In our comparisons, propagation helps only over the video graph, and fusing the modalities in one classifier, or adding inertial edges to the graph, performs worse.