Adaptive method improves concept removal in image diffusion models
GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models
Computer Vision and Pattern Recognition
Summary
Text-to-image AI models sometimes learn unwanted or sensitive ideas from the data they were trained on. The authors propose a new way to remove such concepts without ruining the model’s ability to create good images. Their method carefully targets only the problematic parts of the model and uses automatic tools to avoid manual work. They also add a technique to control how much the model changes during use so it doesn’t lose its original skills. Tests show their approach removes unwanted concepts better while keeping image quality higher than prior methods.
What this means in practice
- •For ai model developers: Remove unwanted or sensitive concepts from pretrained text-to-image diffusion models with minimal impact on image quality and generation fidelity.
- •For content moderation teams: Use improved erasure techniques to reduce generation of inappropriate or non-compliant images while retaining overall model usefulness.
Authors
Qinghui Gong, Yihuai Liang, Yuanlun Xie, Deepak Kumar Jain, Vitomir Štruc, Zhengchun Zhou
Abstract
Text-to-image (T2I) diffusion models inevitably internalize sensitive or non-compliant concepts from large-scale pretraining data, necessitating post-hoc concept erasure. However, existing erasure methods often lack explicit constraints on parameter updates, leading to over-intervention and unintended semantic drift. In addition, many methods rely on manually crafted counterfactual supervision, such as surrogate prompts, which incurs substantial data construction costs that limit scalability to new concepts. To address these limitations, we propose GRACE, a structured concept erasure framework designed to enable localized and selective intervention. Specifically, we introduce a semantically weighted sensitive subspace estimation to precisely lock intervention directions, and employ lightweight subspace-constrained adapters to prevent global semantic disturbance. To eliminate the dependency on manual prompt engineering, we design an automatically decoupled safe-anchor mechanism. To mitigate semantic drift induced by excessive intervention, we introduce an energy-driven dynamic gating mechanism that adaptively controls the timing and strength of intervention at inference. Extensive experiments demonstrate that our method achieves a superior balance between erasure effectiveness and generation fidelity. Compared with the average performance of five state-of-the-art (SOTA) concept erasure methods, our method improves the fine-grained NSFW reduction rate by $17.86\%$, while reducing the macro-averaged target CLIP Score and preservation-oriented Fréchet Inception Distance (FID) by $4.75\%$ and $50.58\%$, respectively, indicating stronger concept suppression with substantially improved preservation of the original model's generative utility.