Diffusion models learn stepwise programs for flexible image editing
Dancing Diffusion: Learning Continuous Programs for Image Generation and Editing
Artificial IntelligenceGraphics
Summary
Diffusion models create images by gradually removing noise, but usually follow one fixed instruction throughout the process. The authors developed Dancing Diffusion, which breaks the image generation into parts, each guided by different instructions acting like steps in a program. This allows changes to be made in later steps without undoing earlier parts, making editing more precise and preserving parts of the image. Their approach improves how well changes stick and keeps more of the original scene intact compared to standard methods. This work opens the door to more structured and controllable image generation.
What this means in practice
- •For graphic designers: Enable step-specific control in image editing workflows to refine and preserve parts of images during generation.
- •For software developers in creative tools: Create image generation features that allow partial re-rendering and localized edits without restarting the entire process.
Authors
Cui Enhe, Jinyue Yang, Bojian Yin, Guoqi Li
Abstract
Diffusion models generate samples through a multi-step denoising trajectory, yet condition every step on the same instruction, so the trajectory cannot be addressed at the level of individual operations. We introduce Dancing Diffusion, which assigns each span of timesteps a dedicated continuous condition block and thereby turns conditioning into a latent program. A compiler maps text features into this program, and a fixed pretrained generator executes its blocks over successive sampling intervals. The trajectory can then dance, keeping its executed history while its continuation is redirected. A shared prefix is executed once and reused, and a later operation is rewritten while content fixed by earlier operations is preserved. The compiler is trained through the images it produces, with visual feedback on both the requested change and the retained content. On SANA, reusing an executed prefix preserves markedly more of the scene than regenerating from noise at a similar strength of edit, learned conditions preserve more background than native prompt switching from the same saved state, and the learned program improves generation quality over native SANA. Later branches weaken the edit, which leaves semantic control over individual operations limited. By converting a single-instruction trajectory into a program of addressable operations, Dancing Diffusion lays a foundation for structured intermediate computation in diffusion models.