FlowAvatar tracks full-body personalized avatars on XR headsets
FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Tracking people’s body, face, hands, and eye movements together in real-time is hard, especially on consumer virtual reality headsets. The authors developed FlowAvatar, a system that creates realistic, personalized avatars by using only sparse sensors on devices like the Meta Quest Pro. Their system is accurate and smooth, and it runs fast enough for live use. They found that having both full tracking and personalized looks together makes the avatars feel more real and natural to users.
What this means in practice
- •For xr developers: Build real-time personalized avatars with complete body and face tracking for VR and AR experiences on standalone consumer headsets.$Commercial implications: Enables creation of lifelike avatars for social and gaming XR applications selling enhanced user presence.
- •For healthcare technology teams: Use full-body and facial tracking avatars to support remote therapy or rehabilitation sessions with natural patient monitoring on consumer XR devices.
Authors
Chenfeng Gao, Taeyoung Yeon, Sungheon Park, Vasco Xu, Anish Prabhu, Mar Gonzalez-Franco, Karan Ahuja
Abstract
Prior avatar embodiment research has studied tracking completeness (how faithfully the body, hands, and face are sensed and reproduced) and personalized appearance in isolation, each requiring specialized lab equipment. How these factors interact on consumer hardware remains an open question, because no existing system unifies body pose, face, hands, and personalized appearance on a standalone headset. We present FlowAvatar, a system enabling real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression, and eye gaze tracking from sparse egocentric inputs on consumer XR devices (Meta Quest Pro). Our pose model achieves competitive spatial accuracy and state-of-the-art temporal consistency on the AMASS benchmark against prior works with comprehensive spatial and temporal losses validated through architecture and loss ablations, while running in real-time. Two user studies (N=12, N=10) provide converging evidence that tracking completeness and personalized appearance serve complementary roles: tracking completeness broadly improves perceived motion quality, while only their combination achieves positive total embodiment.