Generative models trade feature learning against mode collapse risks

The balance between feature learning and collapse in dynamical systems-based generative models

Artificial IntelligenceMachine Learning

Summary

Continuous-time generative models, which create new data by simulating dynamical systems, can sometimes get stuck producing limited results, called mode collapse. The authors used math tools from dynamical systems and operator theory to understand why this collapse happens. They found two main reasons: some collapsed models remain stable under learning, and smaller details take longer to learn, causing an effective collapse when training time is limited. Their work reveals how learning features and avoiding collapse are balanced in these models.

What this means in practice

  • For machine learning engineers: Improve training strategies for continuous-time generative models by managing the trade-off between learning detailed features and avoiding mode collapse.
  • For ai model evaluators: Use insights from operator-based theory to diagnose and predict stability issues in generative models under different training budgets.

A theory result. No direct application yet.

Authors

Julian Brandon, Bruno Loureiro, N Alex Cayco-Gajic, Arthur Pellegrino

Abstract

Over the past decade, a growing body of work has established that implicit biases in learning dynamics fundamentally shape the solutions found by neural networks, governing their learned features and generalisation performance. However, despite these strides in deep learning theory, far less is known about representation learning in generative models based on dynamical systems, from normalising flows to diffusion. Importantly, empirical evidence suggests that continuous-time generative models can be prone to mode collapse, yet a learning theory explaining such biases remains elusive. To address this, we use tools from dynamical systems and operator theory to express the gradient flow of a general class of stochastic differential equations as a composition of linear operators, allowing us to relate collapse to low-rank biases in generative models. In variational inference and flow-matching tasks, we show that this bias can manifest in two different mechanisms: i) where collapsed models are stable fixed points of the gradient flow, or ii) in sequential feature learning, where smaller modes take progressively longer to be learned, producing an effective collapse for a fixed training budget. Together these findings point to a trade-off between feature learning and collapse, suggesting an operator view as a step towards a unified framework for understanding the learning dynamics of dynamical systems-based generative models.