Image source attribution for fine-tuned stable diffusion models
Intra-Architecture Source Attribution for Stable Diffusion: A New Task with Benchmark Dataset and Baseline
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Identifying which AI model created a given image is an important challenge in AI. This paper studies how to tell apart images made by different versions of the same base model, specifically fine-tuned versions of Stable Diffusion XL. The authors collected a large dataset of images from these fine-tuned models and developed an initial method that uses image style cues and specialized network components to attribute images to their source. This approach helps in distinguishing models that share the same architecture but differ in their fine-tuning.
What this means in practice
- •For content moderation teams: Identify which specific fine-tuned stable diffusion model created an image to support provenance and trust verification.
- •For digital forensics analysts: Trace images back to exact fine-tuned versions of diffusion models for investigations into image origins and authenticity.
Authors
Ashutosh Dayal, Mallika Garg, Debashis Sen
Abstract
Designing systems that attribute a generated image to its source model has lately become a major area of AI research. Most existing datasets and systems related to image source attribution are targeted towards the detection of architectural signatures of the source models. Distinct from this, focusing on stable diffusion, we introduce a novel task, IntrAA-SD, Intra-Architecture source Attribution for Stable Diffusion, where the source models to be differentiated share the same architectural backbone and differ only due to fine-tuning. The proposed task aligns with the emergence of fine-tuned generators having the same architecture, like Juggernaut XL and RealVis XL, which are fine-tuned versions of Stable Diffusion XL (SDXL). We provide a first-of-its-kind $140$K-image benchmark dataset containing images generated from different fine-tunes of SDXL, along with in-domain and out-of-distribution prompts. We also provide a baseline framework for IntrAA-SD, which leverages a style-conditioned Mixture-of-Experts network. It uses a frozen CLIP backbone to capture the image style and gate the MoE, which processes fine-tuned trace features from a high-resolution LoRA-adapted DINOv2 network. The fine-tuned models used, the prompting strategies considered, and the formation details employed in the novel dataset are elaborated. Our baseline intra-architecture source attribution method is evaluated in terms of attribution accuracy on our dataset, and ablation studies are also performed. Code and dataset is available at https://github.com/mallikagarg/IntrAA-SD.