PvSync improves lip sync timing and mouth shape matching accuracy
PVSync: A Unified Lip-Sync Expert for Timing and Articulation
Computer Vision and Pattern Recognition
Summary
Matching lip movements with speech sounds is tricky because the timing can be right even when the mouth shapes (articulations) don't match. The authors created PVSync, a tool that checks both if the lips move in time with the sounds and if the mouth shapes fit the spoken sounds. It uses automatic labels grouping sounds that look similar on the lips, which means no manual work is needed. Their tests show PVSync agrees with human opinions on lip syncing better than previous tools and can spot mismatched audio and video accurately.
What this means in practice
- •For video production teams: Automatically assess and improve lip-sync quality in talking-head videos generated for advertising or media content.
- •For augmented reality developers: Enhance virtual avatars by synchronizing mouth movements with speech audio more accurately in real time.
Authors
Kevin Stephen, Varun Menon, Timo Mertens, Nikita Drobyshev
Abstract
Lip movements can match the timing of speech without matching the spoken sounds. We introduce PVSync, a unified model for audio-visual offset estimation and phoneme-level articulation scoring. PVSync combines window-level contrastive learning for synchronisation with a phoneme-level articulation objective that aligns audio and video embeddings of the same viseme class across clips. Visemes group phonemes with similar visible articulation. Viseme labels are derived automatically from forced-aligned transcripts, without manual annotations. On offset-corrected videos from 13 talking-head video generation models, PVSync matches human rankings of lip-sync quality more closely than LSE-C, achieving a Spearman correlation of 0.83 versus 0.34. On an automatically constructed benchmark from held-out speech, PVSync distinguishes viseme-matched from mismatched audio-visual pairs with an ROC AUC of 0.91. PVSync also outperforms SyncNet and MTD-VocaLiST in temporal offset recovery on held-out in-the-wild clips. Code and benchmark data will be released upon acceptance.