Large scale dataset reconstructs detailed 3d human motion from dual viewpoints
Ego-Exo4D Human Meshes Dataset: 4D Human Motion Reconstruction for Ego-Exo Captures
Computer Vision and Pattern RecognitionRobotics
Summary
Tracking how people move in videos from different camera angles is important but difficult. The authors worked with a big video collection that shows people from both a first-person view and from several cameras looking at them. They created a new dataset that adds detailed 3D models of people moving over time, making it easier to study how humans act and learn skills. They also shared the code to create these 3D motion models, helping others work with this data.
What this means in practice
- •For computer vision engineers: Use the dataset and pipeline to generate detailed 3D human motion models from synchronized egocentric and exocentric video captures.
- •For game developers: Create realistic character animations from multi-view human motion data captured in natural settings for immersive gameplay.
Authors
Abhiram Maddukuri, Georgios Pavlakos
Abstract
Ego-Exo4D is a large-scale dataset providing synchronized egocentric and multi-view exocentric video, a rich resource for skill learning and assessment, procedural activity understanding, and embodied AI. However, the dataset ships with only sparse 3D human pose annotations, and reconstructing dense human motion from its multi-view captures is nontrivial. To this end, we present Ego-Exo4D-HM, a large-scale dataset of 4D human motion reconstructions for Ego-Exo4D's captures, and release the accompanying reconstruction pipeline. The code, dataset, and documentation can be found at https://abhiram824.github.io/egoexo4d_human_meshes.