Frequency aware diffusion models improve personalized image style and color transfer
Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating personalized images by combining text and reference photos can be tricky because different parts of the image (like foreground and background) get mixed up during the creation process. The authors found that this problem happens because of how different detail levels (frequency bands) are combined. They made a method called Dual-FDM that separates these detail levels to better handle two tasks at once: changing the style of certain parts and transferring colors correctly. Their approach improves the quality and accuracy of personalized images compared to earlier methods.
What this means in practice
- •For graphic designers: Create customized images combining specific foreground styles and background colors more accurately for marketing and content creation.
- •For video game developers: Generate personalized character skins and environments by better transferring style and color details from multiple reference images.
Authors
Haipeng Liu, Yang Wang, Meng Wang
Abstract
Personalized image generation aims to synthesize text-driven images conditioned on reference images, while mainly casting the generation as image customization for foreground and style transfer for background. Previous arts of diffusion models suffers from the text misalignment with background for image customization and foreground for style transfer during the denoising process. Such facts, as we observed, rooted from the entanglement among hybrid frequency bands during the denoising process. To address such salient limitation, in this paper, we study personalized generation based on dual references - customization and color and style reference - and propose a paradigm to disentangle these Dual image references within Frequency-aware Diffusion Models, dubbed Dual-FDM, to simultaneously tackle two crucial personalized image generation tasks: customization style transfer and color style transfer, by disentangling different frequency bands via mask strategy within frequency domain. For customization style transfer, we replace the mid-frequency band of the background in the style reference with that from the foreground of the customized reference. For color style transfer, we substitute the low-frequency band of the background in the style reference with that from both the foreground and background of the color reference. Both the substituted frequency bands are used as the key and value to reconstruct the query foreground and background of the denoised personalized image.Extensive experiments validate the superiority of Dual-FDM over the state-of-the-art diffusion models for personalized image generation. Our code can be accessed from https://github.com/htyjers/Dual-FDM.