DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors studied how to better spot fake videos that trick both sight and sound. They used many pre-trained models to look at different clues like mouth movement, facial expressions, and audio emotions instead of relying on just one type of feature. By combining these clues with a special method called Mixture-of-Experts, their system was better at detecting deepfakes across different datasets compared to other methods. Their approach showed improved accuracy in multiple tests and they shared their code publicly.
deepfake detectionaudio-visual analysispre-trained modelsMixture-of-Expertsfacial expressionsmouth movementshead posecross-domain evaluationfeature extractionfake video detection
Authors
Vlad Hondru, Florinel Alin Croitoru, Iuliana Georgescu, A. Sophia Koepke, Radu Tudor Ionescu
Abstract
Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We conjecture that overfitting can be mitigated by extracting multiple high-level cues from the available audio and visual modalities via pre-trained models. We therefore assemble a wide variety of pre-trained models to extract features that encode mouth movements, face parsing, facial expressions, head pose, gaze tracking, heart rate, audio emotion and speech activity. We further integrate both unimodal and multimodal cues via a Mixture-of-Experts (MoE) backbone to detect deepfakes. We perform in-domain and cross-domain experiments on five benchmarks for deepfake detection (MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) to compare our framework (DF-MoE) with state-of-the-art methods. Our results indicate that DF-MoE obtains superior deepfake detection results, surpassing all competing methods. We release our code at https://github.com/vladhondru25/DF-MoE.