Dreaming learning helps keep generative models diverse and stable
Mitigating Model Collapse through Dreaming Learning
Machine Learning
Summary
Generative models can lose diversity and perform worse when trained on their own synthetic data, a problem called model collapse. The authors studied this problem by looking at the math behind how the models explore possible outputs. They found that sampling with lower randomness makes models more deterministic and more prone to collapse, but training on their own outputs created with higher randomness helps the model stay diverse. This approach, called Dreaming Learning, keeps the model's internal math balanced and prevents collapse, even for large language models like GPT-2.
What this means in practice
- •For ai developers: Train generative models with Dreaming Learning to maintain output diversity and improve stability over long training on synthetic data.
- •For nlp engineers: Use Dreaming Learning to reduce degeneration effects when fine-tuning large language models like GPT-2 on synthetic or generated text.
Authors
Matteo Benati, Alessandro Londei, Denise Lanzieri, Raffaello Mastromarino, Vittorio Loreto
Abstract
Training generative models on synthetic data causes loss of diversity and a decrease in performance, a phenomenon known as model collapse. We study this degeneration spectrally in a controlled Markov setting. The eigenvalue spectrum of the learned transition operator offers a direct view of the process, since the subdominant eigenvalues set how fast the chain mixes and therefore how much of the state space it continues to explore. We find that sampling at temperature $T_s < 1$ acts on the operator like repeated sharpening, pushing it toward a deterministic system: the subdominant eigenvalues approach the unit circle and the spectral gap narrows. When $T_s < 1$, even data accumulation does not appear to arrest this drift. We then analyze Dreaming Learning, a regulariser that, in parallel with regular training, adds a loss term on the model's own samples drawn at a raised temperature $T_d > 1$. Because high-temperature samples flatten the predicted distribution, training on them keeps the entries of the learned operator from becoming vanishingly small, which in turn keeps the eigenvalues bounded away from the unit circle. Measurements of the spectra over 5000 generations are consistent with this picture, and the mitigation effect persists in more complex settings such as LSTM language models and a 124M-parameter GPT-2.