Papers for

virtual avatar developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Real-time lip syncing with emotion control for all character styles

Emo-LiveSync: Real-Time Universal Lip Synchronization with Explicit Emotion Control

Abstract: Lip synchronization, the task of generating realistic lip movements synchronized with audio signals, is fundamental to applications such as film dubbing, virtual avatars, and digital content creation. Recent GAN-based methods offer computational efficiency but often lack visual fidelity, while diffusion-based approaches achieve high-quality results at the cost of substantial computational overhead. Moreover, most existing methods are limited to photo-realistic human faces, restricting their applicability to stylized characters and non-human entities. In this paper, we propose an efficient universal lip-sync video generation framework that achieves high-quality results without compromising computational efficiency. We build on LivePortrait, a warp-based portrait animation model, and introduce a training strategy that aligns audio signals to an implicit motion space by distilling knowledge from a pre-trained video-to-motion model. This enables real-time inference while maintaining high visual fidelity, and naturally generalizes to diverse stylistic characters beyond photo-realistic humans. Furthermore, we introduce an expressive control gate that enables explicit manipulation of emotional expressions for content enrichment. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both visual quality and lip-sync accuracy while ensuring real-time inference.

Fri 6 NovComputer Vision and Pattern Recognition
The gist
Making characters move their lips so they match what they are saying is tricky, especially if you want it to look good and work fast. The authors improved a way to do this by teaching a computer to understand the sounds and turn them into natural lip movements in real time. Their method works not just for realistic human faces but also for cartoon or stylized characters, and it lets you control the character’s emotions as they speak. This makes creating videos and animations with talking characters easier and more expressive.
Open → 2611.09074v1

Objective metrics better predict human judgments for co speech gestures

Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

Abstract: Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations. Unlike Semantic Score (SC), which shows no significant association with the evaluated perceptual dimensions, SGP is selectively aligned with speech-aware human judgments. We construct five target-specific composite metrics aligned with the subjective dimensions. These composites improve perceptual alignment across all five dimensions, with the largest gains for absence of animation errors and content match, indicating that complementary objective signals can better approximate human judgments than individual metrics alone. Overall, our results show that objective metrics require validation against subjective evaluations.

Thu 8 OctComputer Vision and Pattern Recognition
The gist
Evaluating computer-generated gestures that accompany speech is tricky because existing measurements don't always match what people actually perceive. The authors created a new way to judge these gestures by combining various computer-based measurements with feedback from real users. They introduced a new metric that tracks how well the meaning of gestures is kept in generated movements. Their tests show that some new combined metrics better match what humans think, especially in timing and content accuracy. Overall, they emphasize the importance of checking automatic scores against human opinions.
Open → 2610.11669v1

Pixel conditioned video diffusion enables streaming head avatar animation

PixReenact: Pixel-Conditioned Causal Video Diffusion for Streaming Head-Avatar Reenactment

Abstract: Streaming head-avatar reenactment aims to animate a reference image according to a live driving video, requiring robust motion transfer, long-term identity stability, and low latency. Existing methods often rely on specialized identity or motion representations, which can discard useful visual information and inherit failure modes from external extractors. In addition, many recent diffusion-based reenactment methods use offline, clip-based generation, jointly processing and denoising an entire video clip before producing its output, making continuous low-latency streaming difficult. We introduce PixReenact, a pixel-conditioned streaming reenactment framework built on causal video diffusion. PixReenact conditions directly on VAE-encoded reference and driving frames, without specialized identity or motion representations. To separate reference identity from driver motion, we train with cross-identity pseudo supervision together with corrective objectives anchored to the original reference and driving inputs. Long self-rollouts reduce autoregressive drift, while state-aware dual-teacher distillation separately addresses cold-start and steady-state generation. Across three cross-identity benchmarks and a long-horizon streaming benchmark, PixReenact demonstrates robust cross-identity reenactment, particularly under challenging conditions such as extreme viewpoints, occlusions, and pronounced facial expressions, while maintaining the reference identity over long streams. A 4-NFE rolling student continuously emits four frames per update with a mean emission latency of 239 ms.

Sun 4 OctComputer Vision and Pattern Recognition
The gist
Animating a still image of a person's head in real time based on live video input is hard because it requires keeping the person's identity stable while accurately copying movements, and it must happen quickly. The authors developed PixReenact, a new way to do this using a video diffusion model that looks at raw pixels instead of abstract motion or identity codes. Their method works continuously with low delay, handles difficult cases like unusual angles and facial expressions, and keeps the person's look consistent over long periods. This can help animate avatars smoothly as a person moves live.
Open → 2610.05233v1

Audio lip-sync method recreates person specific mouth details

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

Abstract: We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.

Tue 29 SeptComputer Vision and Pattern Recognition
The gist
Most current systems that animate talking faces produce generic mouths that do not match the real person's unique lip and teeth details. The authors propose RGOR, which improves this by using reference images of the person’s own mouth to guide the animation. RGOR is trained with a special judge model that checks if the generated mouth matches the person's real mouth, not just any plausible mouth. Tests show RGOR keeps the person's mouth details while syncing talking animations accurately.
Open → 2609.38019v1

Reliable co speech gesture generation balancing semantics and rhythm

When Semantics Matter: Reliability-Aware Semantic-Rhythm Control for Co-Speech Gesture Generation

Abstract: Co-speech gesture generation aims to synthesize natural gestures that are both temporally synchronized with speech and semantically consistent with the spoken content. Although recent methods can generate rhythmically plausible motions, they often rely heavily on acoustic prosody while underutilizing textual semantics, especially when semantic annotations are incomplete, noisy, or unavailable. Consequently, the generated gestures may follow speech rhythm while failing to express the intended semantics. To address this problem, we propose a reliability-aware semantic-rhythm control framework for co-speech gesture generation. We first learn a discrete motion prior that represents continuous gestures in a compact and structured motion-code space. We then introduce a dual-branch semantic contribution estimation mechanism consisting of a full multimodal branch and an audio-only branch. Their distributional discrepancy is formulated as conditional information gain to quantify how much textual semantics changes the predicted motion. Based on this estimate, a controllable semantic-rhythm objective selectively strengthens semantic guidance in content-relevant segments while limiting unnecessary semantic intervention in rhythm-dominant segments. Furthermore, we treat background noise as an acoustic reliability condition and introduce noise-conditioned feature modulation together with beneficial latent perturbation to improve generation robustness under realistic acoustic environments. Experiments on benchmark datasets demonstrate that the proposed framework achieves a favorable balance among semantic expressiveness, rhythmic synchronization, motion diversity, and robustness, enabling reliable and controllable co-speech gesture generation.

Tue 29 SeptComputer Vision and Pattern Recognition
The gist
Making animated gestures that match spoken words can be tricky because gestures should flow with the speech rhythm and also show the meaning behind the words. The authors found that past methods rely too much on sound patterns and not enough on the actual meaning of speech, which can cause gestures to miss important content. They designed a new system that watches how much meaning influences motion and adjusts the gestures accordingly, improving how well gestures match both meaning and rhythm. Their system also handles noisy sound better, so it works well even in less-than-ideal listening situations.
Open → 2609.36685v1