EgoFound3R improves egocentric hand reconstruction accuracy sixfold faster
EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes
Computer Vision and Pattern Recognition
Summary
Understanding how hands move in first-person videos is hard because cameras and hands often block each other. The authors created EgoFound3R, a single model that simultaneously figures out the 3D shape of hands in the real world and how each part interacts with objects. Their approach combines different design elements so it works faster and better than older methods. Tests show it reduces errors a lot and predicts detailed hand interactions without extra steps.
What this means in practice
- •For augmented reality developers: Create more accurate and faster hand tracking in AR headsets by integrating world-space hand geometry and interaction details in one model.
- •For robotic manipulation teams: Improve robot understanding of human hand positions and contacts from first-person cameras to better assist or mimic human actions.
Authors
Hongming Fu, Jingcheng Shi, Wenjia Wang, Binhua Zuo, Bo Zhao
Abstract
Egocentric video has become a primary source of supervision for embodied models, and its value rests on recovering hand motion in world coordinates, which camera motion and hand occlusion make difficult. Existing reconstruction pipelines typically separate hand and scene estimation, leave interaction attributes to separate task-specific models, and invoke several models per video, so no prior reconstruction model estimates these attributes and throughput becomes a practical constraint on large-scale annotation. We therefore introduce EgoFound3R, a unified end-to-end model that estimates world-space hand geometry in a metric scale shared with the scene, and predicts point-wise interaction attributes, including visibility, contact, and distance. The model integrates three designs: (i) structured hand prompts that transfer pretrained geometric priors to world-space hand reconstruction; (ii) an explicit hand representation that decodes hand geometry and interaction attributes; and (iii) a shared-parameter multi-rate design that lowers inference cost. Together, these designs predict hand geometry and point-wise attributes in one pass. On OakInk-v2, TACO, and HOI4D, EgoFound3R reduces the mean per-joint position error (MPJPE) by 43.2%, 22.4%, and 11.6% over previous methods and predicts point-wise contact and distance alongside the geometry in the same pass, while attaining approximately 6x higher throughput.