Audio video translation boosts video deepfake detection accuracy
Cross-modal Translation via Conditional Latent Denoising for Video Deepfake Detection
Artificial Intelligence
Summary
Detecting fake videos created by altering both sound and visuals is hard because the two parts are connected in subtle ways. The authors came up with a method that helps the computer translate information between audio and video signals to spot inconsistencies caused by manipulation. This approach looks at how the audio and visual parts relate inside the computer’s hidden understanding, making detection better. Tests show their method can find deepfakes more reliably by comparing the two signals carefully.
What this means in practice
- •For video security teams: Improve detection tools by integrating audio and video signals to better identify deepfake content in surveillance or media monitoring.
- •For media authentication services: Add a cross-modal consistency check to authentication workflows to catch subtle manipulations in audio and video streams.$Commercial implications: Enables more reliable commercial products verifying multimedia authenticity for journalists, legal cases, and content platforms.
Authors
Xinzhe Li, Youzhi Tu, Kong Aik Lee
Abstract
The growing threat of video deepfakes necessitates multimodal detection. Beyond serving as independent indicators of authenticity, audio and visual signals have intrinsic dependencies that also provide an essential criterion for detection. Previous methods often overlook the cross-modal correspondences, hindering information transfer between domains and leaving crucial detection cues unexplored. To address this challenge, we propose a framework called Cross-modal Translation via Conditional Latent Denoising (CTCLD) for video deepfake detection. It connects the distinct distributions of heterogeneous modalities in latent spaces, enabling smooth cross-domain information transfer to improve detection performance. We first establish a Bayesian foundation by decomposing the audio-visual joint distribution. Subsequently, CTCLD translates both modalities via bidirectional latent denoising conditioned on each other, effectively capturing subtle inconsistencies in the manipulated signals. Experimental results demonstrate that the proposed CTCLD enables comprehensive domain alignment, resulting in a robust video deepfake detection approach with competitive performance.