System for precise hand and head tracking from egocentric video
MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild
Computer Vision and Pattern Recognition
Summary
Tracking hand and head movement in videos is hard, especially outside controlled studios. The authors created MEgoVista, a method that uses video from a wearable camera to accurately measure hand and head motion in real-world settings. It ensures measurements use real-world sizes and separates the wearer's hands from others around them. They tested their method against a standard motion capture system and found it reliable for collecting motion data outside the lab.
What this means in practice
- •For augmented reality developers: Enable accurate measurement of users' hand and head movements from wearable cameras for realistic interaction in AR applications.$Commercial implications: Allows creation of AR products with precise real-world scale hand tracking from portable devices, improving user experience outside controlled environments.
- •For sports coaching teams: Capture detailed hand and head motion data from players wearing simple cameras during training without specialized motion capture setups.
Authors
Jiangong Xiao, Zhihao Zhang, Yifei Dong, Chao Ma, Zhouyi Jin, Zhiwen Hou, Li Liu, Weihuang Chen, Hongbin Sun, Maoqing Yao
Abstract
Learning manipulation from human video requires high-fidelity hand-motion reconstruction in metric units. Today's metric hand labels come from studio rigs and instrumented headsets, and both are confined in the same two ways: neither leaves a prepared setting, and neither is checked against an independent reference. Unconstrained head-worn recording promises the opposite trade-off, scaling with the number of people wearing a device. We therefore introduce MEgoVista, an offline pipeline that turns a single unprepared MEgo View recording into metric two-hand and head motion in one gravity-aligned world frame. Three properties set it apart from existing egocentric reconstruction systems: first, it reconstructs in settings studio volumes and tabletop rigs cannot reach, settling hand ownership at detection so bystander hands stay out of the wearer's trajectory; second, it takes its metric gauge from calibrated stereo rather than a monocular prior, installing scale at initialisation so policies receive physical units, not arbitrary coordinates; third, both outputs are scored inside a motion-capture volume against independent Chingmu optical capture, under a protocol that audits its own reference and charges what a method declines to predict. MEgoVista is offered as a measured route from egocentric video to metric hand supervision, one that widens where such labels can be gathered.