Egoxperience enables precise world-level 3d scene and action reconstruction
EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
Computer Vision and Pattern Recognition
Summary
Understanding what a person is doing from a video taken from their own eyes is tricky because everything—the camera motion, the environment, and hand-object actions—needs to fit together in a real-world space. The authors created EgoXperience, a complete approach with new data, automatic labeling, and a model that learns to reconstruct these elements together in one system. Their method uses 50 hours of videos showing tasks indoors and outdoors with detailed annotations for hands, objects, and locations. With EgoXperience, they achieve far better accuracy than past methods in knowing where the camera, hands, and objects are in 3D space.
What this means in practice
- •For augmented reality developers: Create more accurate AR experiences by reconstructing a user’s environment and hand-object interactions in real-time from a first-person camera.
- •For robotics engineers: Improve robot perception and human-robot collaboration by understanding detailed human actions and surroundings from wearable cameras.
Authors
Xinhao Ji, Hui En Pang, Hao Li, Kairui Hu, Jiner Wang, Mingze Sun, Ziang Cao, Yalun Dai, Thang Luu, Qin Ma, Hongyi Chen, Yukang Cao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
Abstract
Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world. Achieving this coherence requires both joint reconstruction and aligned supervision, yet existing methods and datasets offer limited support for either. We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking. Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects. This yields EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion within and across indoor and outdoor environments, with second-level task captions. We carefully select one hour across these activity categories as EgoX-Bench to evaluate joint reconstruction. To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets. Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass. Extensive experiments demonstrate state-of-the-art performance across scene estimation and world-space hand-object pose recovery. Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0%.