Parts-aware guidance improves audio-driven gesture portrait videos
PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation
Computer Vision and Pattern Recognition
Summary
Creating videos where a person's gestures match their speech well is difficult. The authors present a new method called PAGE that helps generate more accurate and realistic co-speech gesture videos from just one picture and an audio clip. Their approach focuses on important body parts like lips, eyes, and hands to improve visual detail, and uses smart checks during video creation to keep gestures and speech in sync. They also introduce a large Chinese speech video dataset to better test these methods. Experiments show PAGE beats previous approaches in quality and gesture variety.
What this means in practice
- •For digital content creators: Generate realistic talking portrait videos that match speech audio with improved lip and hand gesture accuracy from a single image and audio clip.$Commercial implications: Enables sale of software tools for personalized video messages and virtual spokespersons with higher realism.
- •For game developers: Create more lifelike in-game characters that produce synchronized speech and gestures based on dialogue audio without manual animation.
Authors
Yabiao Wang, Jiafu Wu, Yicheng Xu, Teng Hu, Shengzhe Zhou, Yanxiao Sun, Jiangning Zhang, Ying Tai, Yong Liu, Ming-Hsuan Yang
Abstract
Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal. Despite advances in visual quality, diffusion-based approaches still suffer from degraded local details and imperfect audio-visual alignment. We present PAGE (Parts-Aware GuidancE), a general, backbone-agnostic optimization framework for diffusion-based generation that enhances local spatial fidelity and global audio-visual consistency at both the training-objective and denoising-trajectory levels. During training, we propose Parts-Specific Reweighting (PSR), which adaptively increases supervision in semantically important regions such as lips, eyes, and hands, improving visual fidelity in perceptually critical areas. During inference, we introduce Consistency Classifier Guidance (CCG), an audio-conditioned latent classifier that identifies and corrects audio-visual inconsistencies in the generated video. We further design Differential Guidance (DG), an efficient inference strategy that refines facial and body motion separately while avoiding cross-region interference. To enable large-scale evaluation, we construct the Chinese Announcer Speech (CAS) dataset, the first comprehensive Chinese benchmark for co-speech gesture portrait video generation. Compared with the widely used PATS dataset, CAS includes substantially more identities, video clips, and scene diversity. Extensive experiments on both PATS and CAS demonstrate that PAGE achieves state-of-the-art performance in perceptual quality, gesture diversity, and audio-visual alignment. Project page: https://aigc-explorer.github.io/PAGE/.