DiffSafeMerge: Mitigating Backdoor Inheritance in Diffusion Model Merging
2026-08-10 • Cryptography and Security
Cryptography and SecurityComputer Vision and Pattern Recognition
AI summaryⓘ
The authors study a problem where merging pre-trained image generation models can secretly introduce harmful backdoors that don't show up during normal use. They propose a method called DiffSafeMerge (DSM) that uses a small set of clean images and special tests to detect and reduce suspicious parts in the merged model without hurting image quality. They tested this on multiple attacks and datasets, showing that DSM effectively blocks these backdoors while preserving good image generation. Compared to other methods that prevent attacks, DSM also keeps image quality better on average.
Diffusion modelsModel mergingBackdoor attacksDenoising lossAttack mitigationImage generationAdversarial robustnessFID scoreUnlabeled data
Authors
Jiayang Zhang, Ji Guo, Jiachen Li, Wenshu Fan, Wenbo Jiang
Abstract
Unconditional diffusion checkpoint merging assumes benign sources, yet a compromised public checkpoint can transfer a dormant backdoor while clean generation appears normal. Mitigation is difficult without knowing the compromised source, trigger, or target, and broad sanitization may degrade image quality. We introduce DiffSafeMerge (DSM), which uses a small unlabeled clean set and fixed, attack-agnostic stress probes to score source blocks, shrink suspicious contributions toward a trusted reference, and select attenuation under a clean denoising-loss budget. We evaluate four attacks, two datasets, and 21 target conditions. Intended merging already has zero worst-target ASR in 10 of 14 source cases; DSM preserves these outcomes and records no target match in the remaining four over three seeds, including three with baseline ASR of 48--100\%. Among methods with zero worst-target ASR on both datasets, DSM obtains the lowest case-averaged FID in the matched seed-0 comparison.