Dreamhmr improves human body and hand motion capture from videos
DreamHMR: From Video Generation to Human Motion Modeling via Structural Focusing
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Capturing detailed human body and hand movements from videos is difficult, especially when parts are hidden or blurry. The paper introduces DreamHMR, a method that uses video generation technology to better understand human shapes and movements over time. It first captures the overall body pose and then refines details of the body and hands using clearer local information. The authors show that DreamHMR works better than previous methods across several datasets, especially when parts are hard to see.
What this means in practice
- •For augmented reality developers: Improve real-time human motion capture in AR apps by integrating detailed body and hand movement estimation robust to occlusion.$Commercial implications: Enables creation of more convincing AR avatars and interactive experiences by delivering accurate full-body and hand motion tracking.
- •For animation studios: Enhance animation pipelines with better automated 3D human motion capture from everyday video, reducing manual keyframing effort.
Authors
Zhengjie Xu, Ruihao Gong, Xianglong Liu
Abstract
Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.