Fast and accurate 3d hand and camera tracking from egocentric video
V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos
Computer Vision and Pattern Recognition
Summary
Understanding hand movements seen from a wearable camera is hard because the camera itself moves too. The authors created V-JEPA Hand, a model that simultaneously figures out where the hands and camera are in 3D space from video clips. Their model processes a clip in one go, without breaking the task into steps, making it simpler and faster than previous methods. It also works well on standard test sets and runs at nearly 100 frames per second on good hardware.
What this means in practice
- •For augmented reality developers: Integrate real-time 3D hand and camera pose tracking from wearable cameras to improve AR experiences with accurate virtual object interaction.
- •For robotics engineers: Use fast, accurate 3D hand and camera trajectory estimation from egocentric video to teach robots human-like hand movements and task execution.
Authors
Ruicheng Li, Qixiu Li, Yu Deng, Sicheng Xu, Jiaolong Yang, Baining Guo
Abstract
Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model achieves the lowest coverage-aware mean per-joint position error (MPJPE) among comparable baselines on ARCTIC, HOT3D, and EgoStandard. It also yields lower world-frame trajectory errors than the compared SLAM-based pipelines. Ablations support the benefits of video pretraining, temporal context, and mixed supervision. V-JEPA Hand achieves a forward-pass throughput of 93.4 FPS on a single RTX A6000. Project page: https://qft-333.github.io/V-JEPA-Hand-Page/.