Self-Regulating Annealing in Heavy-Tailed Diffusion Models
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study diffusion models, which are used to generate data, but standard ones assume data follows a normal (Gaussian) distribution. They improve this by using a Student's t-distribution, which better handles data with heavy tails (extreme values). They design a new sampling method based on stochastic differential equations that adjusts noise depending on the current state. This adaptive approach helps the model better reproduce heavy-tailed data, and the authors verify its effectiveness through theory and experiments.
Diffusion modelsGaussian distributionStudent's t-distributionHeavy-tailed distributionStochastic differential equationsSamplingNoise scaleAnnealing mechanismDeep generative modeling
Authors
Keito Wakatsuki, Hideaki Shimazaki
Abstract
Diffusion models have emerged as a leading framework for deep generative modeling. While the standard Gaussian formulation is theoretically convenient, its suitability for heavy-tailed datasets remains unclear. To address this, heavy-tailed diffusion models (HTDMs) extend the standard formulation by replacing the Gaussian distribution with a Student's t-distribution, thereby improving tail fidelity on heavy-tailed datasets. Although stochastic differential equation (SDE)-based sampling is possible in HTDMs, it has not been fully explored. In this paper, we propose an SDE-based sampler for HTDMs that explicitly incorporates a state-dependent diffusion coefficient. This state dependence naturally induces a self-regulating annealing mechanism by adaptively modulating the effective noise scale. We theoretically explore this mechanism and experimentally verify its necessity for reproducing samples from a heavy-tailed distribution.