Deepfake detectors improve accuracy using new visual model and experts
Unifying Semantic Priors and High-Frequency Traces: Enhancing V-JEPA with Mixture-of-Experts for Robust Synthetic Image Forensics
Computer Vision and Pattern Recognition
Summary
Fakes images on social media can spread false information, so detecting them is important. The paper shows that a type of AI model called JEPA helps recognize real versus fake images better by understanding how the world typically looks. The authors built a new system, MoE-JEPA, which uses multiple expert parts and attention techniques to better spot fake details, even in tricky cases like compressed images. Their method tested on a large benchmark and beat bigger existing models in accuracy.
What this means in practice
- •For social media platforms: Detect manipulated images more reliably even after compression or transmission to reduce misinformation spread.
- •For digital forensics teams: Identify AI-generated or tampered images in large multimedia collections with higher precision using the new forensic model.
Authors
Simone Teglia, Irene Amerini
Abstract
The unchecked proliferation of manipulated images on social media platforms has increased the spread of misinformation, posing a severe threat to public trust and information integrity. Modern deepfake detectors typically rely on Vision Transformers (ViTs) to capture the low-level inconsistencies that characterize fully synthetic or locally tampered images. However, the global understanding of such foundation models is not enough to discriminate alone between real and fake multimedia content, especially in challenging scenarios where images are compressed or transmitted through social media. In this paper we pioneer the application of Joint-Embedding Predictive Architecture (JEPA) models to deepfake detection, taking advantage of the generalized representation of visual reality that such World Models have exhibited. We hypothesize, and empirically demonstrate, that the intrinsic world understanding of JEPA models can be used as a strong prior for a deepfake detector. To fully exploit JEPA capabilities, we propose MoE-JEPA, a dual-stream architecture for deepfake detection. By enhancing a V-JEPA 2 backbone with a Residual Mixture-of-Experts (MoE) mechanism, along with a noise stream branch, our model dynamically internalizes forensic knowledge. Furthermore, a Gated Attention Multiple Instance Learning (MIL) module is employed to ensure precise spatial semantic understanding. Evaluated on the SID-Set benchmark, comprising 300K AI-generated, tampered and authentic images, MoE-JEPA establishes a new state-of-the-art with an accuracy of 95.54%, successfully outperforming vastly larger models.