Realis dataset improves AI image detector testing and reliability
REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Detecting AI-generated images is hard because models often rely on easy clues that don’t work well with new image generators or after images are edited. The authors introduce REALIS, a large and carefully chosen collection of real and AI-made images from many different text-to-image tools, which helps test detection methods more fairly. They also offer a special set called REALIS-Expert, with very similar real and fake images, and a way to test how detectors handle various image edits. Their tests show that models trained on REALIS detect AI images more reliably, especially on difficult, processed images.
What this means in practice
- •For security teams: Evaluate and improve AI image detectors to better identify fake content even after heavy image editing or from new generation tools.
- •For media verification teams: Use the dataset to benchmark tools that verify real images in news and social media to catch manipulated or AI-generated visuals.
Authors
Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya, Artem Filippov, Sergey Lavrushkin, Dmitriy S. Vatolin, Anastasia Antsiferova
Abstract
AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or processed images. Existing datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana 2. REALIS combines prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot vision-language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.