Optimization based method improves 3D animal motion capture from video
ORMA: Optimization-based Monocular 4D Reconstruction of Articulated Animals
Computer Vision and Pattern Recognition
Summary
Capturing the detailed 3D shape and movement of animals from regular videos is very hard because animals come in many shapes and it’s tricky to get accurate 3D data for training. The authors show a new way to do this without needing prior training by separating an animal’s pose from its 3D body shape and improving details with optimization. They also created a new synthetic dataset to test how well their method works. Their approach produces better, consistent 3D models and tracks animal motion more reliably across different species.
What this means in practice
- •For visual effects artists: Create detailed and accurate 3D animal models and animations from simple video footage for films or games without extensive manual modeling.
- •For wildlife monitoring teams: Track and analyze animal movements in the wild using only video cameras to understand behaviors across different species more accurately.
Authors
Xuyi Hu, Francesco Palandra, Shangzhe Wu, Daniel Cremers, Riccardo Marin, Silvia Zuffi
Abstract
Recovering articulated 4D representations of animals from monocular videos remains challenging due to the large diversity of quadruped morphologies and lack of animal 4D supervision data. Existing learning-based reconstruction methods operate on individual images and rely on synthetic or model-fitted 3D supervision, which inherits the constraints of strong parametric priors and limits generalization to out-of-distribution species. When applied to out-of-distribution animals, they often recover a plausible pose while producing inaccurate geometry because the underlying shape model cannot faithfully represent the observed instance. We present ORMA, a training-free reconstruction framework that decouples articulation from shape, using the predicted pose as reference for optimization while leveraging generative 3D priors for accurate shape reconstruction. Given a reference image, we reconstruct the animal geometry and register it to the parametric model SMAL+, yielding an articulated shape adapted to the observed instance. We then combine per-frame articulated pose estimates with globally consistent camera poses to recover animal motion in a shared world coordinate frame, and further refine the reconstruction using self-supervised DINO correspondences and temporal consistency. To enable quantitative evaluation, we introduce PAW4D, a synthetic multi-species benchmark with ground-truth 3D geometry and camera motion. Experiments on PAW4D, PFERD, and challenging in-the-wild videos demonstrate that ORMA improves reconstruction accuracy while recovering globally consistend animal motion across diverse quadruped species.