OmniPoser recovers full body motion using flexible sensor data
OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching
Computer Vision and Pattern Recognition
Summary
Recovering how a person moves in 3D from a few wearable sensors is hard because these sensors can be noisy, blocked, or unstable in real life. The authors developed OmniPoser, a system that works with different kinds of sensor inputs like motion trackers or head-mounted devices, all at once in a single model. Their design uses a special way to handle missing or mixed data and a fast approach to generate full human motion quickly. OmniPoser works as well as current best methods but is much faster, running at over 700 frames per second.
What this means in practice
- •For virtual reality developers: Recover full body movements from various sensor combinations to enhance immersive user experiences with accurate real-time avatars.
- •For sports coaching teams: Track athletes’ complete 3D motion from sparse sensors to analyze performance in real-world outdoor settings quickly.
Authors
Minghao Liu, Tutian Tang
Abstract
Recovering 3D human motion from sparse wearable sensors is challenging due to the inherent noise, occlusion, and instability of real-world tracking signals. Existing approaches are restricted to fixed sensor configurations and, when built on diffusion models, are computationally prohibitive for real-time deployment. We propose OmniPoser, a universal framework that recovers full-body SMPL motion from a flexible combination of heterogeneous inputs, such as 3D keypoints, sparse IMU rotations, head-mounted 6-DoF poses, or any subset thereof, within a single model. OmniPoser introduces a cross-modal masking mechanism paired with a Masked Conditioning Encoder, enabling one architecture to incorporate diverse sensor configurations through a masked design. A decoupled dual-stream generative backbone generates complete motion via Conditional Flow Matching, requiring only a single ODE integration step at inference. Across three settings and seven benchmarks, including in-the-wild Nymeria and Ego-Exo4D datasets, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS. The project page is available at https://jamesbridgewater51.github.io/omniposer-eccv/.