Facial parts analysis improves face forgery detection accuracy

FaceJev: Direct Visual Decisions for Face Forgery Detection

Computer Vision and Pattern Recognition

Summary

Detecting fake faces often works well on known fake types but struggles with new ones. The authors created FaceJev, a method that checks eight different facial parts separately to decide if a face is fake. They made a big dataset with labeled facial part anomalies to train this. This approach links which part looks abnormal directly to the fake detection, resulting in better accuracy on new fakes. Their tests show FaceJev performs significantly better than previous methods.

What this means in practice

  • For digital security teams: Detect previously unseen face manipulations in security video systems by analyzing individual facial parts for abnormalities.
  • For social media platform engineers: Improve detection algorithms to flag manipulated profile images by integrating direct visual assessment of specific facial features.

Authors

Siran Peng, Tianshuo Zhang, Yibo Xue, Ajian Liu, Haoyuan Zhang, Weisong Zhao, Shuhuan Chen, Xiangyu Zhu, Zhen Lei

Abstract

Face forgery detection methods often perform well on seen manipulations but generalize poorly to unseen ones. Multimodal Large Language Models (MLLMs) can recognize facial parts and assess their visual plausibility. Existing MLLM-based methods typically express these judgments through textual explanations, but seldom treat individual part decisions as explicit learning targets. We present FaceJev, an MLLM-based method for face forgery detection through direct visual decisions. Following the Jev formulation, FaceJev answers eight independent questions about facial parts using a shared image encoding and reads anomaly probabilities from the native answer logits. A face is classified as forged if any part is predicted to be abnormal, directly linking the detection result to the part decisions. We construct PartForensics, a dataset of approximately 160k images with binary anomaly targets for the eight facial parts, to supervise these decisions. We further propose FS-GRPO, which optimizes each part decision through an exact reward comparison with its alternative, using both detection and part rewards. FaceJev outperforms the strongest baseline by 7.25 percentage points on unseen manipulations. Extensive ablation studies validate our design choices and the effectiveness of the main components.