Diffusion models lose some features faster when trained on synthetic data

Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training

Machine LearningArtificial Intelligence

Summary

When AI systems called diffusion models learn from data generated by earlier versions of themselves, they can lose important details, a problem called model collapse. The paper compares two training methods: one where real data is completely replaced by synthetic data, and another where training uses a fixed amount of mixed real and synthetic data, even as more data accumulates. The authors find that completely replacing data leads to rapid loss of features, while the fixed-budget approach retains some features for many generations. They also provide a mathematical model to explain why some features disappear quickly and others last longer depending on the method.

What this means in practice

  • For ai model engineers: Maintain feature diversity in diffusion models by using fixed-size mixed data training rather than fully replacing real data with synthetic data.
  • For image recognition system developers: Design data pipelines that reduce degradation of key image features during iterative training with synthetic datasets.

Authors

Hanna Malet, Gabriel Turinici

Abstract

Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.
View PDFOpen arXiv