Links between diffusion models and optimal transport in data generation

Some remarks on the connection between generative diffusion and optimal transport

Machine Learning

Summary

This work explains how a process used to create data, called generative diffusion, is connected to a mathematical way of matching distributions called optimal transport. The authors show that adding noise to data and then removing it can be understood through an entropic optimal transport problem, with the noise level controlling how smooth the process is. They also study how distinct groups in data appear during generation, comparing diffusion methods with flow-based ones. Finally, they reveal a new perspective on how the reverse part of the diffusion process can be seen as minimizing a cost related to entropy changes.

What this means in practice

  • For machine learning engineers: Design improved generative models by using insights on how noise level controls optimal transport regularization during data synthesis.
  • For statistical model designers: Analyze mode formation in complex data distributions using the detailed connection between diffusion, flow matching, and optimal transport.

A theory result. No direct application yet.

Authors

Giulio Biroli, Marc Mézard

Abstract

We discuss three simple connections between generative diffusion and optimal transport. First, the joint law of the data and of their Ornstein-Uhlenbeck noising is, at every time, the solution of an entropic optimal-transport problem, at a temperature set by the noise level. Backward diffusion can consequently be read as an annealing which gradually decreases the entropic regularization. Second, we study the speciation transition--how the modes of the data distribution emerge during generation--and compare how it happens in diffusion and flow matching with respect to optimal transport. Using Gaussian mixtures as high-dimensional example, we show that the dynamical instability through which diffusion and flow matching separate the modes becomes, for optimal transport, an exponentially thin boundary layer at the beginning of generation, which is rounded by entropic regularization. Third, considering a generic forward dynamics with a general confining potential, we show that the backward process driven by the score also satisfies a drift-based variational principle whose minimal cost is the entropy difference between initial and final distributions.
View PDFOpen arXiv