Avatar operator verification improves by combining face motion and video
Unveil the Mask: Jointly-Optimized Diffusion Masked Pretraining for Appearance-Agnostic Operator Verification
Computer Vision and Pattern Recognition
Summary
Recognizing who is controlling a photorealistic avatar is hard because the avatar’s appearance hides the operator’s identity. The authors address this by focusing on how the face moves rather than what it looks like. They use two types of information: the positions of key facial points over time and the changes between video frames. By training a system to understand these motions and combining both sources, the method better identifies the operator across different avatars and speaking styles without needing any outside pretrained models.
What this means in practice
- •For security system developers: Create operator verification tools that reliably identify users controlling avatars despite changes in avatar appearance.
- •For game developers: Enhance player identity verification in games using avatars by analyzing face motion rather than appearance.
Authors
Katsuya Ogata, Ryoya Yoshiyama, Kawachi Hodaka, Yuta Nakashima
Abstract
Photorealistic avatar videos decouple the appearance of a rendered face from the person who operates it, which makes conventional face recognition unsuitable for identifying the operator. Verifying the operator therefore requires representations of how a face moves that are insensitive to what the avatar looks like. We propose a verification framework built without any external pretrained model. It uses two complementary modalities provided for each sample: facial landmark sequences and video frames. From the landmark modality, it learns motion representations by diffusion masked pretraining over grouped landmarks, guided by lightweight auxiliary tasks that predict behavioral cues such as motion intensity and eye openness, and turns them into operator embeddings with a contrastive linear probe. To generalize to unseen avatar generators, the pretraining input is randomly transformed to simulate how a generator may alter the operator's motion. From the video modality, it captures fine facial motion through differences between consecutive frames. The embeddings of the two modalities are combined into one. Our approach reaches an overall AUC of 0.769 on the WBA2026 benchmark. Linear probing shows that the video embedding retains little information about the avatar's identity, whereas the landmark embedding retains some, and fusing the two modalities improves verification in particular across speaking and listening modes.