ExMR improves hand and foot motion capture from a single video
ExMR: Extremity Motion Refinement for Monocular Human Motion Capture
Computer Vision and Pattern Recognition
Summary
Capturing detailed hand and foot movements from a single video is very hard. The authors created a system called ExMR that first estimates the whole body's motion and then refines the detailed movements of hands and feet using extra clues from the video. Their method makes the captured motions of fingers and ankles more accurate than previous approaches. This helps create more realistic animations and better motion analysis from ordinary videos.
What this means in practice
- •For animation studios: Create more detailed and accurate hand and foot movements from single-camera video for realistic character animation.$Commercial implications: This system can be sold as a tool to improve animation quality from easily acquired video footage.
- •For sports analytics teams: Analyze athletes’ detailed hand and foot motions from standard video to better understand technique and performance.
Authors
Tom Wehrbein, Bodo Rosenhahn
Abstract
Detailed hand and foot articulation is important for animation and motion analysis, yet remains challenging to recover from monocular video. Recent expressive video methods infer hand articulation from shared whole-body representations. Our conference work suggests that this underexploits local evidence: the Foot Motion Refinement (FootMR) module refines ankle rotations from foot keypoints, improving reconstruction even when the initial estimator receives the same keypoints. We extend this principle to hands with the Hand Motion Refinement (HandMR) module. Because 2D finger keypoints are often missing or unreliable, HandMR combines image features of a pretrained hand reconstruction model with initial arm and hand rotations to refine elbows, wrists, and fingers. We integrate FootMR and HandMR into ExMR (Extremity Motion Refinement), a system for reconstructing 3D body, hand, and foot motion from monocular video. An initial temporal estimator predicts whole-body motion, after which both refiners revisit local evidence to predict residual joint rotations. Experiments on ARCTIC, MOYO, RICH, and MOOF show that ExMR outperforms state-of-the-art methods in hand and foot reconstruction, reducing hand per-vertex error and ankle joint angle error on MOYO by 19% and 18%, respectively, relative to the strongest competitor. Code will be made available for research purposes. Project page: https://twehrbein.github.io/exmr-website/