Diffusion models reveal when image features form during generation
Feature Information Dynamics in Diffusion
Artificial IntelligenceMachine Learning
Summary
Diffusion models create images by gradually removing noise, but it wasn’t clear when specific details appear during this process. The authors developed a way to measure when certain features start to form by using information theory and denoising errors. They showed that image features appear in a certain order, from coarse to fine, and this differs across types of image representations. This helps explain how these models build up images step-by-step and suggests ways to improve training.
What this means in practice
- •For computer vision engineers: Improve training of image diffusion models by using insights into the order of feature generation to design better conditioning strategies.
- •For machine learning platform developers: Develop diagnostic tools that track when features form during generative model training to better monitor and debug diffusion models.
Authors
Jia-Shu Pan, Tao Zhang, Yufei Huang, Yanjun Sheng, Tailin Wu
Abstract
Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class $\to$ mask $\to$ Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at https://github.com/AI4Science-WestlakeU/feature-information-dynamics.