Deepfake detection improves with pulse and face movement analysis
Beyond Ambiguous Visual Cues: Studying Physiological Disruptions and Cross-Modal Inconsistencies in Deepfake Videos
Computer Vision and Pattern Recognition
Summary
Deepfake videos are fake videos often created by swapping or changing faces in videos. The authors show that these fake videos disrupt natural body signals like heartbeats seen in face color changes. They created a method that looks at both the heartbeat signals and facial movements together to better spot fakes. Their approach performs better than methods looking at just one of these signs and works well even on new datasets.
What this means in practice
- •For video security teams: Detect manipulated face videos by jointly analyzing heart pulse signals and facial movements in surveillance or online content.
- •For mobile app developers: Integrate improved deepfake detection models that combine physiological signals with facial features to verify video authenticity on smartphones.$Commercial implications: Enables selling verification apps or services with enhanced detection of face-swapped videos leveraging joint physiological and behavioral cues.
Authors
Chenxi Yang, Yassine Ouzar, Larbi Boubchir
Abstract
Recent deepfake detection studies increasingly suggest remote photoplethysmography (rPPG) signals as an authenticity cue. However, existing benchmarks lack physiological ground truth, and current detectors underexplore the cross-level relationship between facial features and physiological dynamics, often relying on late fusion or rPPG features alone. In this paper, we construct high-fidelity deepfake manipulations on established real rPPG datasets (COHFACE and UBFC-rPPG) to investigate how forgeries disrupt natural physiological signals and facial behavior at the same time. Building on this analysis, we propose a bidirectional co-attention fusion detector that jointly models rPPG and facial behavior tokens. This mechanism explicitly captures the cross-level dependencies between pulse dynamics and facial motion to learn a robust, joint authenticity representation. Extensive experiments using a subject-disjoint 5-fold evaluation demonstrate the superiority of our approach. Achieving a 92.80\% AUC on constructed datasets using face swapping and 96.78\% AUC on motion transfer, our model outperforms both the rPPG-only single modality baseline and the best feature-level fusion methods. Furthermore, transfer-learning result of the fusion detector on Celeb-DF-v2 while keeping both feature extractors fixed achieves 91.20\% accuracy and 86.08\% AUC, which suggests applicability under target-domain adaptation.