Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models
2026-05-11 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address the challenge of removing certain concepts, like copyrighted or explicit content, from large text-to-image models. They found existing quick methods less effective for bigger models, so they created SPACE, which changes specific parts of the model's attention system to erase unwanted ideas. SPACE focuses these changes in a smaller, important area, making the erasure more accurate and using less memory. Their tests showed that SPACE works better and is more efficient than older methods.
text-to-image diffusion modelsconcept erasurecross-attentionStable Diffusionclosed-form updatesparsitymodel parametersadversarial promptsmemory efficiency
Authors
Nicola Novello, Andrea M. Tonello
Abstract
Erasing specific concepts from text-to-image diffusion models is essential for avoiding the generation of copyrighted and explicit content. Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques, but they become less effective when scaling from smaller models such as Stable Diffusion 1.5 to larger models like Stable Diffusion XL. To maintain erasure effectiveness in these larger-scale architectures, we propose SParse cross-Attention-based Concept Erasure (SPACE). SPACE iteratively modifies the cross-attention parameters of a model with a closed-form update that jointly induces sparsity and erases target concepts. By concentrating the concept mapping to a lower-dimensional subspace, SPACE achieves superior erasure efficacy compared to dense baselines. Extensive experimental results show improvements in erasure effectiveness and robustness against adversarial prompts. Furthermore, SPACE achieves 80\%-90\% cross-attention sparsity, reducing the storage requirements for saving the modified parameters by 70\%, demonstrating its memory efficiency.