Locality constrained diffusion guides zero shot image enhancement across domains
Domain-adaptive Zero-Shot Image Enhancement via Locality-Constrained Diffusion Guidance
Computer Vision and Pattern Recognition
Summary
Images from one type of source, like paintings or blurry ultrasound scans, often need to be improved to look realistic in another style, such as photos or high-quality medical images. Existing methods can change important local details too much or fail to make images look real enough. The authors created a method called LocDiff that carefully keeps important local features intact while enhancing less critical parts. This works without needing example pairs of before-and-after images, making it practical for challenging tasks like turning art into photos or improving ultrasound images.
What this means in practice
- •For medical imaging teams: Improve low-quality ultrasound images by enhancing resolution and realism without needing paired training data.
- •For digital artists and photo editors: Convert paintings or other artistic images into photorealistic photos while preserving key identity features like facial details.
Authors
Theresa Neubauer, Dimitrios Lenis, Astrid Berg, Maria Wimmer, Gaia Romana De Paolis, Philip Matthias Winter, David Major, Johannes Novotny, Ariharasudhan Muthusami, Katja Bühler
Abstract
Denoising Diffusion Probabilistic Models have shown remarkable performance in unconditional image generation. In order to generate images with desired semantics, recent works have restricted the solution space by using guidance constraints in the diffusion sampling process. However, for image enhancement across different domains, these methods struggle to balance two main requirements: looking realistic in the target domain (photorealistic images) and preserving relevant features of the source domain, e.g., low-quality renderings or art paintings. Here, small local changes can alter the fidelity of the image completely, while large changes in other regions might be insignificant. We introduce LocDiff, a locality-constrained guidance method for image enhancement, which serves as a zero-shot extension to pre-trained diffusion models, ensuring the preservation of critical features during domain adaptation. In this way, we retain important local features, while allowing less critical regions to remain unconstrained and not interfere with the guidance process for relevant regions. We evaluate our method on two different domain-shift tasks: For art-to-photo translation, we apply the method in a fully zero-shot setting, preserving facial identity from paintings while generating photorealistic details. For enhancing low-quality fetal ultrasound renderings, we demonstrate zero-shot inference with auxiliary prior alignment. Here, the objective is to artificially add high-resolution characteristics and produce photorealistic ultrasound renderings, a target domain for which no ground truth distribution exists. Our experimental results demonstrate that LocDiff achieves favorable realism-faithfulness trade-offs compared to state-of-the-art methods, enabling controllable cross-domain enhancement.