Egocentric video enables joint 3d human motion and scene reconstruction
EgoMo3R: Joint Egocentric Motion and Scene Reconstruction
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Capturing how people move and what their surroundings look like in 3D from a video recorded on their head is hard, especially because head movements are quick and blurred. The authors introduce EgoMo3R, a system that guesses both the person's 3D movements and the environment's shape at the same time from regular videos. It uses visual hints like depth maps and motion from the video, plus a model of how humans usually move, to improve both predictions. Testing on challenging datasets shows their approach works better than others focused on only one task.
What this means in practice
- •For augmented reality developers: Build AR experiences that accurately track a user's body and surroundings using only a single head-mounted camera without additional sensors.$Commercial implications: Enables AR devices to provide realistic user interaction and environment mapping from monocular egocentric video, improving immersion and usability.
- •For roboticists: Improve robots' ability to understand and navigate environments by reconstructing human motion and scene geometry from wearable cameras during human activities.
Authors
Chaitanya Patel, Yusuke Kato, Rin Takahira, Kazuki Kozuka, Juan Carlos Niebles, Ehsan Adeli
Abstract
Understanding holistic 3D human motion and the surrounding environment from egocentric video is fundamental for applications in AR/VR and robotics. Although these tasks are inherently coupled, existing methods treat them separately. In particular, egocentric motion reconstruction methods typically assume access to metric, gravity-aligned camera trajectories from specialized inertial sensors. Recovering such trajectories from monocular egocentric video remains a significant challenge due to rapid head movements and motion blur. We present EgoMo3R, a joint optimization framework that simultaneously reconstructs 3D human motion and scene geometry from casual egocentric video. It leverages the robustness of low-level visuo-geometric primitives, including monocular depth, optical flow, and point tracks, integrated with a diffusion-based human motion prior via first-order optimization. We propose an alternating optimization scheme that establishes a synergistic feedback loop: scene reconstruction provides physical grounding and trajectory conditioning for the motion prior, while the motion prior regularizes the camera trajectory and anchors the scene in a metric, gravity-aligned world frame. We evaluate EgoMo3R on the highly dynamic sequences of the EgoExo4D and Nymeria datasets and show that it outperforms specialized baselines in both 3D scene reconstruction and egocentric motion estimation.