Full-body pose tracking improved with multiview RGB cameras in VR
MuCHMDPoser: Multiview RGB-Camera Full-Body Pose Tracking for Virtual Reality
Computer Vision and Pattern Recognition
Summary
Tracking the entire body in virtual reality is hard when only using standard gear like headsets and hand controllers because these devices mainly track upper-body movements. The authors show that adding multiple regular cameras around the user helps track the lower body much more accurately. They also developed a quick calibration process that doesn’t need any special markers and runs smoothly at 60 frames per second using a good graphics card. This system improves how people’s full poses are represented in virtual spaces.
What this means in practice
- •For virtual reality developers: Improve avatar lower-body motion accuracy in VR apps by integrating multiview RGB camera pose inputs alongside standard tracking devices.$Commercial implications: Enables VR companies to create more immersive and realistic full-body experiences, enhancing VR games and social platforms with better motion capture.
- •For motion capture studios: Use low-cost RGB cameras to enhance body pose precision without requiring bulky marker setups during VR motion capture sessions.
Authors
Antoine Maiorca, George Fletcher, Thierry Ravet, Thierry Dutoit
Abstract
Accurate full-body tracking in Virtual Reality (VR) is heavily underconstrained when relying solely on consumer-grade sparse sensors, such as Head-Mounted Displays (HMDs) and handheld controllers. Indeed, the self-avatar full-body pose must be inferred based on the 3 available upper-body joints tracked by these devices. This leads to significant tracking error, especially in lower-body pose. To overcome this issue, we propose to extend the tracking setup with external multiview RGB cameras. We show that integrating RGB-based pose estimation this framework is able to significantly decrease lower-body tracking errors, even under common artifacts in RGB-based estimates. Additionally, we evaluate a only-once HMD-camera calibration procedure that does not require any physical markers. We finally propose an implementation of the overall framework, achieving an end-to-end animation at 60 FPS on a Nvidia RTX3080 GPU. Codes are available at https://github.com/antmaio/MuCHMDPoser.