Domain manifold unlearning improves selective concept removal in diffusion models
Compositional Concepts Erasing: Disentangling Object-Style via Domain Manifold Unlearning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Diffusion models, which create images from descriptions, can accidentally produce unwanted or copyrighted content. The authors developed Domain Manifold Unlearning (DMU), a method that helps these models forget specific object-style combinations without losing general abilities. DMU uses features inside the model to identify and erase only the unwanted parts while keeping other knowledge intact. This approach also reduces the risk of the model forgetting too much when removing several concepts one after another.
What this means in practice
- •For ai content moderation teams: Remove harmful or copyrighted content from diffusion models while preserving their overall image generation quality.
- •For digital artists and designers: Control diffusion models to selectively erase specific styles or objects without degrading their general creative capabilities.
Authors
Yang Xiao, Kaiyuan Deng, Gen Li, Zinan Ling, Zhiwen Fan, Xiaolong Ma, Bo Hui
Abstract
Diffusion models pose significant risks regarding the generation of illicit content and copyright violations. This has catalyzed the development of machine unlearning to erase harmful concepts or copyrighted styles from a model's generative repertoire without compromising its general performance. However, previous unlearning approaches on diffusion models have largely focused on either object-level or style-level removal and failed to address the challenge of object-style combination unlearning. At the same time, multiple unlearning requests often lead to catastrophic forgetting. To address these challenges, we propose Domain Manifold Unlearning (DMU), which uses an EMA-updated feature proxy extracted from intermediate diffusion features to represent the retained target-object direction. As a lightweight estimating representation of an object, DMU enables selective unlearning of domain-specific components in a specific object while preserving other semantic representations, thereby significantly enhancing robustness in multiple combination unlearning scenarios. Extensive experiments show that DMU significantly mitigates catastrophic forgetting, outperforming state-of-the-art methods of batch unlearning and sequential unlearning.