Improved deepfake detection with shared and private audio visual features
Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection
SoundArtificial Intelligence
Summary
Detecting fake videos or audio is harder when the data comes from many different sources with different tricks. The authors created a method that looks at shared clues across all sources as well as unique clues from each source to better spot fakes. They tested their approach on two collections of deepfakes and found it catches tricky cases more reliably than earlier methods. This helps make deepfake detection more fair and accurate even when training examples come from mixed origins.
What this means in practice
- •For security operations teams: Deploy multi-source aware audio-visual deepfake detectors that maintain accuracy across different source distributions and manipulation types.
- •For media verification services: Improve reliability of audio-visual content authenticity checks when inputs come from various origins with uneven attack types.$Commercial implications: Enables more robust authenticity verification services for media companies by detecting subtle deepfake artifacts across multiple data sources.
Authors
Qiwu Wen, Jahangir Alam
Abstract
Audio-visual deepfake detectors are often trained under a single dataset or manipulation regime, but multi-source training collections may contain heterogeneous attacks and uneven source distributions. Training separate detectors for each source is inefficient, while a single shared detector may be biased toward dominant source-specific patterns. We study known-source imbalanced multi-source audio-visual deepfake detection using FakeAVCeleb and LAV-DF, where source identity is supplied during training and inference. To address this setting, we build on the Multi-Scale Cross-Modal Transformer Encoder (MSCT) and propose Pre-MSCT Shared-Private Feature Fusion. Our method routes \texttt{[CLS]}-augmented audio and visual tokens into a shared stream and source-specific private streams before the first MSCT block, while retaining its internal multi-scale self-attention (MSSA), differential cross-modal attention (DCA), and feed-forward operations. The shared stream models cross-source reusable evidence, while the private streams retain source-dependent residual variations. A domain-aware gated residual fusion module injects selected private evidence into the shared representation for the main fused prediction. Experiments under two controlled source-composition protocols show improved macro and worst-source AUC compared with the MRDF and MSCT baselines. These results support shared-private cross-modal discrepancy learning in the evaluated two-source setting with known source identity.