Egocentric camera system captures full body and hand motion simultaneously
EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations
Computer Vision and Pattern Recognition
Summary
Capturing full body and hand movements from a camera worn on the head is tricky because the camera’s view is limited and hands can get hidden or look different in size. The authors made a system with six wide-angle cameras on a wearable rig to see the body and both hands from different angles. They use smart methods to combine data from cameras to get accurate body and hand positions and movements in real time. Their approach improves accuracy compared to using just one camera and does not need extra cameras pointed from outside.
What this means in practice
- •For virtual reality developers: Create realistic self-avatar movements by capturing full body and hand motion with wearable cameras alone, without external sensors.
- •For robotics teleoperation teams: Use egocentric wearable cameras to accurately capture operator body and hand states for controlling robots remotely in real-time.
Authors
Yaoqing Hu, Xuguo He, Daohan Cui, Yukun Shi, Kai Chen
Abstract
Head-mounted body and bimanual motion capture supports self-avatars, teleoperation, and embodied demonstrations. However, a single egocentric camera is vulnerable to a limited field of view, scale ambiguity, and hand-object occlusion. We present EgoWholeView, a wearable capture system of six synchronized wide-angle RGB cameras with a role-specialized pipeline. The outward views localize the rig in a world frame, while an overlapping downward pair independently predicts SMPL-X body and MANO hand parameters. Both predictors are adapted to egocentric fisheye data. To combine the two, reliability cues measured from the predictions themselves weight the fusion of body and hand joint rotations, while shape is aggregated across views and frames. The pair's baseline fixes metric scale, and triangulation recovers root translation. Relative to monocular prediction, fusion reduces PA-MPJPE by 3.7% for the body and 11.2% for the hands, and temporal refinement lowers angular-acceleration magnitude by 32.2% and 31.0%, respectively. On our six-camera recording, the pipeline produces body and hand states in a shared world frame using benchmark-fitted gates without retuning, and temporal refinement lowers hand acceleration magnitude by 41.08%. Using only RGB cameras and no third-person view, EgoWholeView offers a low-cost, portable route to egocentric interaction data.