Egorayformer improves 3d human pose estimation from head cameras
EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation
Robotics
Summary
Estimating 3D positions of body joints from cameras worn on the head is hard because these cameras see only part of the body with distorted and wide-angle views. The authors propose EgoRayFormer, a method that shares 3D guesses of each joint's location between both camera views and refines these guesses using geometry and learning. This approach better uses the relationship between the two camera images, helping improve the accuracy of 3D pose estimates. Tests show EgoRayFormer outperforms previous methods on standard benchmarks for egocentric body tracking.
What this means in practice
- •For augmented reality developers: Enhance ego-worn AR systems by improving 3D body tracking accuracy using stereo fisheye camera inputs processed with EgoRayFormer.$Commercial implications: Enables more precise user body tracking in AR headsets, improving interaction and immersion, which can be sold to AR hardware manufacturers.
- •For motion capture studios: Increase robustness and accuracy of head-mounted body motion capture setups for virtual production and animation workflows leveraging combined stereo view refinement.
Authors
Haozhe Wang, Songyuan Wu, Jihai Xu, Jie Zhao
Abstract
Head-mounted stereo fisheye cameras provide a compact view of the wearer's body, but strong distortion, partial visibility and a short baseline make 3D pose estimation difficult. Existing refinement decoders sample image features independently in each view, without requiring the samples to correspond to a common 3D hypothesis. We introduce EgoRayFormer, a causal pose estimator that couples stereo feature sampling and geometric refinement in ray space. For each joint, the decoder projects shared 3D hypotheses along and around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects the resulting rays; a learned gate blends this geometric estimate with a learned pose update. This formulation introduces an explicit stereo constraint while retaining flexibility when the rays provide weak depth constraints. In local evaluations on the official device-relative test splits, EgoRayFormer achieves three-seed mean MPJPEs of 17.64 mm on UnrealEgo2 and 49.55 mm on UnrealEgo-RW, improving to 47.57 mm with UnrealEgo2 pre-training. In paired validation ablations, removing the geometric branch increases MPJPE in all three seeds at both input resolutions. With a common ResNet-18 encoder and input resolution, our system reduces RW test MPJPE by 9.09 mm relative to the device-relative EgoPoseFormer baseline. A separate protocol analysis quantifies the difference between device-relative and oracle-root evaluation.