AI generated videos become almost impossible to detect by humans or tools
DF26: We Cannot Tell Fake From Real Anymore
Computer Vision and Pattern Recognition
Summary
People find it nearly impossible to tell if certain videos of someone speaking on camera are real or made by AI. The authors created DF26, a set of videos including hundreds of real clips and thousands made by different AI models, to test detection skills. Both humans and the latest fake video detectors performed no better than guessing. This shows current methods to find fake videos are not reliable anymore with newer AI-generated media.
What this means in practice
- •For video security teams: Assess performance limits of current fake video detectors when analyzing synthetic public-speaking videos from diverse AI models.
- •For media verification services: Evaluate and improve content verification processes for videos in political or official communication using modern synthetic benchmarks.
Authors
Severyn Shykula, Andrii Yermakov, Ivan Samarskyi, Dmytro Mishkin, Jan Cech, Anastasiia Mishchuk
Abstract
We introduce DF26, a novel benchmark for detecting AI-generated videos containing fully synthetic clips produced by recent text-to-video and image-to-video models. The videos capture single-person public-speaking scenarios, spanning direct-to-camera recordings, official statements, and studio interviews - 271 real and 2,420 synthetic videos generated by seven modern video models. The study on DF26 shows that human performance in detecting AI-generated videos, as well as state-of-the-art deepfake detectors, is close to random chance. Our results highlight the limitations of current evaluation protocols and motivate the need for benchmarks that explicitly measure robustness to modern generative model distribution shifts.