Object removal from images without retraining using stable diffusion
Zero-Shot Object Removal via Attention Masking, Latent Anchoring, and Refinement
Computer Vision and Pattern Recognition
Summary
Removing an object from a photo is tricky because you need to erase the object completely and fill the space realistically so it looks natural. This paper shows a way to do that without needing to train a new computer model for each specific task. Instead, the authors use a combination of smart masking, caption understanding, and step-by-step image refinement to remove objects and fill in the background smoothly. Their method works well on complicated scenes and cleans up leftover traces using repeated improvements.
What this means in practice
- •For photo editing software developers: Incorporate zero-shot object removal features that need no task-specific training for more flexible user editing tools.$Commercial implications: Enables building more versatile and user-friendly photo editors that can robustly remove objects without extra training, appealing to broad consumer markets.
- •For digital advertising teams: Use automated object removal in images to quickly create marketing materials without unwanted items disrupting the background scene.
Authors
Arman Taghizadeh, Ulf Krumnack, Kai-Uwe Kühnberger
Abstract
Removing an object from a real image requires more than synthesizing plausible content within a mask: the method must suppress residual object features, preserve the unedited scene, and generate replacement content that is consistent with the surrounding background. This paper approaches object removal from a stage-based perspective and proposes a zero-shot framework for constrained latent inpainting with a frozen pretrained Stable Diffusion model, requiring no task-specific training or model fine-tuning. The method integrates SAM-based mask construction, BLIP image-caption conditioning, DDIM inversion, background-weighted masked null-text optimization, decoder self-attention masking, hard outside-mask latent anchoring, and localized renoise--denoise refinement into a unified pipeline. The method is evaluated through qualitative examples, quantitative local-consistency metrics, and ablation studies. The results demonstrate effective object removal and context-consistent replacement content. The ablations indicate that background-weighted masked NTI is particularly beneficial for structurally complex backgrounds, whereas the no-NTI variant is sufficient in other evaluated examples. Repeated refinement further reduces object remnants and boundary artifacts remaining after the primary editing pass.