The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction
2026-06-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present ViDiHand, a new method to track detailed hand movements using video from a first-person view. Unlike older approaches that struggle with blocked views or rely on limited hand-pose data, ViDiHand uses a pretrained video diffusion model that has learned to understand complex motions from large-scale video. They adapted this model specifically for hands and added a decoder to get precise 3D hand poses directly from full video frames. Their method works better than previous ones on several hand motion datasets without needing extra detectors or optimization during testing.
4D hand motion reconstructionegocentric videovideo diffusion modelhand pose estimationhand-object interactionmotion dynamicsocclusion reasoningARCTIC datasetHOT3D datasetHOI4D dataset
Authors
Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang, Tianyi Wei, Zhiwei Zeng, Siyuan Huang, Zhiqi Shen, Xingang Pan
Abstract
4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.