Classifier-Free Guidance harms controllability in text and image editing
Rethinking Classifier-Free Guidance for TI2I Flow Matching
Computer Vision and Pattern RecognitionMachine Learning
Summary
This paper looks at a technique used in AI image editing called Classifier-Free Guidance, which is usually helpful for making high quality images. The authors found that, surprisingly, this technique can actually hurt the ability to control the editing when used in certain noisy stages of the process. They suggest a simple way to fix this early in the editing steps, which helps keep better alignment between the instructions and the image changes without extra cost. Their fix works well on different methods of image editing based on flows.
What this means in practice
- •For image editing software developers: Improve controllable image editing tools by correcting guidance techniques to better follow user instructions in flow-based models.
- •For digital content creators: Create more precise and user-aligned edited images from text and image inputs using enhanced flow modeling with better early-step guidance.
Authors
Andreas Floros, Elena Koritska, Teofilo de Campos, Christoph Vogel, Chu-Tak Li, Anuj Sharma, Alexandros Neophytou
Abstract
We study Text-and-Image-to-Image (TI2I) flow matching methods for controllable image editing. In a broader generative modeling context, Classifier-Free Guidance (CFG) is established as a key component responsible for high quality outputs. However, we argue that its role in TI2I modeling is poorly understood. Specifically, we make the surprising observation that the CFG mechanism is implicitly harmful for editing in the noisy phase of flow modeling, hindering instruction following and controllability. We propose a simple correction in the early flow steps, enabling a richer alignment-preservation tradeoff at no extra cost. The effectiveness of our approach is demonstrated on both pixel-space and latent TI2I flows.