Facial animation from muscle signals works with vr headsets on
emg2face: Expressive Facial Animation with High-Density Surface EMG
GraphicsComputer Vision and Pattern RecognitionEmerging TechnologiesMachine Learning
Summary
Capturing facial expressions with cameras is hard when people wear VR headsets that cover their faces. The authors show that using high-density sensors on the skin to record muscle signals can predict detailed facial movements instead. They recorded signals from many spots on faces and trained a neural network to turn these signals into 3D expressions. This approach works even when the face is hidden and can animate different human or non-human faces.
What this means in practice
- •For vr hardware developers: Enable realistic facial expression tracking inside VR headsets using skin muscle signals instead of cameras.$Commercial implications: This paper enables VR device makers to include expressive avatar control from EMG sensors embedded in headsets, a capability not feasible with optical methods.
- •For game animation teams: Create real-time facial animations of characters from wearable muscle signals, bypassing the need for complex camera setups.
Authors
Ganidhu Abey, Wendy Greening, Ashika Kamboj, Leonhard Helminger, Abhijeet Ghosh, Karel Petranek, Sergio Orts Escolano, Dinesh K. Pai
Abstract
Facial movements convey subtle and important information that is critical for human social communication. Optical methods for face capture are difficult or impossible to use when the face is occluded by head-mounted devices (HMDs), such as VR headsets. Even with a clear line of sight, such methods raise privacy concerns and require head-mounted capture rigs that offset cameras and lighting from the face. We show that high-density surface electromyography (HD-sEMG) provides a viable non-optical alternative that addresses these challenges. We measured 64 EMG channels, using two textile EMG grids, with 32 from the forehead (typically occluded by an HMD) and 32 from the side of the face. EMG data were digitized at 2048 Hz and filtered. Facial movements were simultaneously recorded and used to estimate 478 3D facial landmarks using MediaPipe's Face Landmarker. A major challenge in such multimodal recordings is synchronizing EMG and video data, which have different sampling frequencies and independent clocks. We developed a novel synchronization method using analog audio bursts that is capable of sub-millisecond synchronization. We also developed a staged fitting method that fits a recent high-resolution parametric head model (GNM), with 253 identity blendshapes and 383 expression blendshapes, to the MediaPipe landmarks as participants performed different facial expressions. We trained a deep neural network comprising per-grid spatial encoders followed by a dilated temporal convolutional network (TCN) to predict blendshape parameters from HD-sEMG signals at 100 Hz. Once trained, the network can predict expression blendshapes solely from HD-sEMG recordings. The output can be rendered using standard real-time blendshape animation methods. We demonstrate the methods using recordings from 25 participants, and direct expression transfer to a variety of human faces and non-human characters.