Modular pipeline improves comparing audio visual deepfake detectors
DFD-Lab: A Modular Audio-Visual Deepfake Detection Pipeline
Computer Vision and Pattern RecognitionSound
Summary
Detecting fake videos that combine audio and visuals is complicated because different methods use different setups and data. The authors built DFD-Lab, a system that keeps data handling, models, and tests separate but worked together, making it easier to compare detectors. They ran tests showing that adding some image distortions during training helps improve detection, but accuracy can still drop on new unseen data. Their work highlights how tricky it is to detect fakes when using different sources and why different evaluation methods matter.
What this means in practice
- •For security operations teams: Use a modular pipeline to fairly compare and improve audio-visual deepfake detection methods under varying conditions.
- •For multimedia platform developers: Integrate interchangeable detectors to maintain content integrity by identifying manipulated audio-visual media across datasets.
Authors
Jan Rybarczyk, Mateusz Roszkowski, Jacek Komorowski
Abstract
Comparing audio-visual deepfake detectors requires coordinating dataset adaptation, temporal input representation, model interfaces and experimental conditions. We present DFD-Lab, a modular pipeline that separates these responsibilities while supporting shared training and evaluation workflows. We integrate three implementations: Xception-based maximum-logit fusion, ResNet with temporal LSTM fusion, and our AVFF reimplementation. Experiments cover external testing, degradation-based training augmentation and evaluation-time corruption. On a filtered subset of Deepfake-Eval-2024, models trained on FakeAVCeleb attain baseline AUROC values of 0.504, 0.538 and 0.458. JPEG50 training augmentation raises these to 0.691, 0.605 and 0.570, respectively, while all three accuracies decrease. These results illustrate why training interventions, evaluation corruptions and metric-dependent outcomes should remain distinct within a common pipeline. The contribution is the integration of audio-visual processing, interchangeable detectors and configurable experimental workflows, supported by empirical case studies. The findings highlight the challenge of cross-dataset detection and the complementary information provided by ranking and classification metrics.