Activation steering technique improves content removal in diffusion models
RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models
Computer Vision and Pattern Recognition
Summary
Sometimes AI models that create images need to stop showing certain things, like unsafe content or copyrighted styles, but still make other pictures well. The authors noticed that old methods accidentally remove parts that belong to things we want to keep, making the pictures worse. They proposed a new way to carefully separate what to remove from what to keep, adjusting the model’s behavior during the image creation process without extra training. Their method works better at removing unwanted concepts while keeping the rest of the images looking good.
What this means in practice
- •For ai content moderation teams: Improve removal of unsafe or inappropriate image elements while keeping overall image quality in AI-generated art platforms.
- •For digital artists and designers: Filter out copyrighted or unwanted art styles from AI-generated images without affecting other creative elements.
Authors
Yongliang Wu, Haori Lu, Yulun Wu, Jinqi Luo, Xingyu Zhu, Yaoyao Liu
Abstract
Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Existing activation steering methods build an erasure direction mainly from the target concept and adjust model activations along it at inference time. However, target and retained concepts often overlap in the model's representation space, so this direction also contains shared components that retained concepts rely on. Steering directly along this direction can therefore suppress retained concepts and harm the generation of non-target content. To address this issue, we propose Retain-aware Activation Steering (RASteer), a training-free method. RASteer first builds a retain subspace from the concepts to preserve. Retain-Orthogonal Steering (ROS) then removes components aligned with this subspace from the erasure direction, making steering more specific to the target. Since fully removing the shared components can weaken erasure, we further introduce Overlap-Adaptive Calibration (OAC). At each layer and denoising step, OAC uses the overlap between the erasure direction and the retain subspace to control how much of each shared component is removed, balancing target erasure and concept preservation. Experiments on unsafe-content, instance, and artistic-style erasure across multiple backbones and benchmarks show that RASteer matches or outperforms the activation steering and weight editing baselines we evaluate, achieving a better balance between erasure and preservation.