FUSED: Forensic-Semantic Mixture-of-Experts for AI Inpainting Detection and Localization
Computer Vision and Pattern Recognition
Summary
The authors created FUSED, a new AI tool that can both detect if an image has been altered using inpainting and show exactly where the changes are. Unlike past methods that only look at the whole picture and miss local edits, FUSED combines detailed clues with bigger-picture understanding to find edits better. It works well even on types of altered images it hasn't seen before and outperforms existing methods on multiple tests. The authors also found that many detectors rely on certain obvious signs that can sometimes mislead them, but FUSED still performs best with or without those signs.
Authors
Anton Nuzhdin, Marcel Worring, Ivona Najdenkoska
Abstract
Diffusion-based inpainting models modify only a localized part of an image, while many AI-image detectors rely on global artifacts and do not localize. These artifacts vary across generators, limiting detector transfer under distribution shifts. Recent work shows that restoring the authentic pixels outside the inpainted region removes these cues and can degrade pretrained detectors. To address this, we present FUSED, a unified framework for the joint detection and localization of AI-generated inpainting. FUSED combines low-level forensic cues with high-level semantic features using a sparsely-gated Mixture-of-Experts architecture, enabling the model to adaptively prioritize the most relevant signal for each token. For each input, FUSED predicts both an image-level manipulation score and a pixel-level mask of the inpainted area. On the OpenSDID cross-generator benchmark, FUSED achieves the best average detection and localization, with the largest gains on unseen generators. The same model transfers directly to the held-out AutoSplice and CocoGlide benchmarks, more than doubling localization performance. Evaluating each held-out benchmark with and without the global generator artifact further shows that all evaluated methods, ours included, partly read the artifact as evidence of manipulation, and FUSED remains the strongest under both conditions. Code and pretrained models are available at https://github.com/AntonNuzhdin/FUSED.