Diffusion model edits images precisely using flexible area control
Overpainting: Localized Context-aware Diffusion Image Editing
Computer Vision and Pattern Recognition
Summary
Editing images can be tricky when you want to change only a specific part without affecting the rest. This paper introduces a method called overpainting that allows users to specify exactly which areas to edit, which may be edited, and which must remain unchanged. The researchers improve a type of AI called diffusion models to do this by teaching it to focus on the right parts using a special mask called a trimap. They also created a way to automatically generate training examples so the model learns many kinds of edits. This approach works well for a variety of image editing tasks, giving users more control.
What this means in practice
- •For graphic designers: Precisely adjust parts of an image while preserving surrounding content for more controlled creative edits.
- •For mobile app developers: Add advanced local image editing controls to photo apps to enhance user editing precision with AI.
Authors
Sam Sartor, Iliyan Georgiev, Michael Fischer, Valentin Deschaintre, Pieter Peers
Abstract
We present "overpainting", an image editing operation which offers both control over the location of the edit and awareness of the previous content in that location. The overpainted area is given by a trimap, where white-annotated pixels must be edited, gray-annotated pixels may be edited, and black-annotated pixels must not be edited. This enables both precise and loose control, depending on user intent. We implement overpainting by adapting a pretrained image editing diffusion model using a combination of joint attention and low-rank adaption across input images with attention-dropout to balance the information flow between noise, source and mask images. We present a novel, automated, training data generation pipeline that (1) generates a set of candidate image pairs leveraging existing language-based editing models, (2) carefully curates those pairs, and (3) extracts a trimap from each usable pair. We demonstrate the versatility of our overpainting model on a wide range of editing tasks.