Physics-based animated characters move realistically in virtual reality
Bringing physics-based controller policies to virtual reality
Human-Computer Interaction
Summary
It’s easy to make things you can touch in virtual reality (VR), but not so easy to have characters move like real people. The authors show that using physics-based control methods, animated VR characters can move in a very lifelike way. They made these methods faster and compatible with popular VR platforms like Unity3D and WebXR. They also created a new way to measure how good the animations look compared to real movement, making comparisons simpler.
What this means in practice
- •For vr game developers: Make VR characters move more realistically by using fast, physics-based control policies compatible with popular VR engines.
- •For mobile vr app developers: Deploy physics-based animation controllers that work on Android via cross-platform compilation, improving character interactivity on mobile VR headsets.
Authors
Joan Llobera, Pierre Nagorny, Balint Hodossy, Valérie Juillard, Caecilia Charbonnier
Abstract
Nowadays, it is easy to set up any element of a VR experience so users can interact with it physically, except for interactive VR characters. Can we extend this to animated VR characters? In this work, we studied whether physics-based character animation (PBCA) controllers can produce high-quality movement in VR-compatible environments (Unity3D, WebXR). We evaluated the quality of the results, as well as the losses at each stage of the processing pipeline: data ingest, training and inference. For data ingest, we computed accuracy metrics related to the retargeting method used. For training, we computed accuracy metrics related to different training policies and different training environments. We developed new JAX implementations of state-of-the-art PBCA controllers (DeepMimic, AMP and ADD) which can be trained in MuJoCo-MJX in minutes, 50 times faster than in a traditional CPU-based training environment. The benefit of these particular implementations is that the resulting control policies can be readily transferred to Unity3D and support cross-platform compilation, including Android, something critical for use in VR production. For inference, we tested these controller policies in Unity3D and WebXR. Our results suggest that DeepMimic and ADD trained in MuJoCo-MJX generate controller policies with better animation fidelity than other methods. To simplify the comparison between controller policies, and different reference animations, we implemented an animation fidelity index (afi). Its results are analogous to other more common metrics, but afi is more intuitive to interpret and thus simplifies the comparison of training methods across the successive stages of the processing pipeline.