Text to image models better preserve subject look with regain correction
ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Text-to-image AI tools are often customized using a few pictures of a person or object, usually taken with a camera. But sometimes these pictures come from the AI itself, which leads to weird colors and too much detail in the final images. The authors pinpointed the problem to a specific step in the AI's image generation called classifier-free guidance. They created ReGain, a fix that adjusts these details during image creation to make the AI-generated images look more like the real subject, without needing any real photos.
What this means in practice
- •For graphic designers: Personalize AI-generated images of clients more accurately using synthetic images without access to real photos.
- •For game developers: Improve fidelity of character visuals when training customization models on AI-generated images instead of real-world photos.
Authors
Shubhang Bhatnagar, Ishan Bhatnagar, Viraj Shah, Narendra Ahuja
Abstract
Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of its images. Increasingly, these images come from a diffusion model rather than a camera. We show that fine-tuning on such synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail. To isolate the cause, we fine-tune two models from the same base model with the same DreamBooth recipe, one on real photos of a subject and one on synthetic images of that subject generated by the first. We trace the degradation to classifier-free guidance (CFG). For the model personalized on synthetic images, the angle between the conditional and unconditional noise predictions, and with it the norm of their difference, is much larger than for the model personalized on real photos. This inflation grows toward high frequencies and also appears at other prompts semantically close to the subject, such as its class noun, but not at unrelated ones. We propose ReGain, a training-free correction applied at sampling time that measures how much each frequency band of the guidance is inflated relative to the base model and scales that band down accordingly. ReGain needs no real photos. On Stable Diffusion v1.5, ReGain closes 51-64% of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I. It also improves subject fidelity on SDXL and SD 3.5 and preserves text alignment on all three backbones.