Detecting fake lip sync videos by spotting lip and head movement mismatches
Ariadne's Thread of LipSync: Unraveling Forgeries via Inconsistency between Lip Motions and Head Poses
Computer Vision and Pattern RecognitionCryptography and Security
Summary
Videos that fake lip movements to match speech are getting very realistic, which can cause problems because it’s hard to tell what’s real. The authors found that real videos naturally link lip movements with head motion, but fake videos miss this connection. They created a method called LipDA that looks for mismatches between lips and head movements to spot fakes and identify which model made them. Their approach works very well on big datasets and can tell fake videos apart with high accuracy.
What this means in practice
- •For social media platforms: Identify and block fake lip-synced videos to reduce misinformation and improve content authenticity.
- •For video forensic teams: Trace the source model of a lip-sync forged video to assist in digital forensic investigations.
Authors
Tianyi She, Jiawei Liu, Weifeng Liu, Hanqing Zhao, Weiming Zhang, Kejiang Chen
Abstract
Recent advances in LipSync generation technology have led to the creation of highly realistic videos, posing severe societal risks. However, existing defense strategies struggle against LipSync forgeries, as advanced LipSync generation methods not only achieve better lip synchronization but also eliminate visual artifacts. An important reason is that they overlook an inherent biological coupling between lip movements and head poses in natural speech videos. In this paper, we propose LipDA, a novel framework for joint LipSync Detection and Attribution, which takes advantage of the inconsistency between head and lip. For detection, the framework learns to quantify this discrepancy by contrasting lip and pose features from authentic versus forged videos. For attribution, our method is designed to capture the unique temporal dynamics and audio-visual synchronization patterns that act as the fingerprint of models, enabling source tracing. We conduct extensive experiments on two challenging LipSync datasets as well as our own proposed large-scale and multi-generator dataset. LipDA achieves over 97\% AUC in detection and 97.5\% accuracy in model attribution, significantly outperforming existing methods. Code and the proposed LipSync-A dataset are available at https://github.com/AnsonShe/LipDA.