Estimating diffusion process scores with shallow neural networks efficiently
Diffusion Score Estimation with Shallow Neural Networks: Regularity, Approximation, and Finite-Sample Rates
Machine Learning
Summary
Estimating how data changes over time in a diffusion process is important for generating new data samples. The authors analyze how well simple neural networks (with one hidden layer) can learn these changes from limited data. They provide mathematical guarantees on the accuracy and reliability of these networks under certain smoothness and growth conditions of the data. This helps in understanding how many samples and how large a network are needed to generate new data distributions closely matching the original.
What this means in practice
- •For machine learning engineers: Design score-based generative models with provable sample efficiency using shallow networks for data generation tasks.
- •For statistical signal processors: Use explicit sample complexity bounds to assess and improve diffusion score estimation methods for noisy signal reconstruction.
A theory result. No direct application yet.
Authors
Fan Liu, Yahong Yang, Wei Zhu
Abstract
We study the sample complexity of estimating Ornstein--Uhlenbeck diffusion scores in $\mathbb R^d$ using denoising score matching (DSM) with finite-width shallow ReLU networks, and its implications for reverse-process sampling. Under weak differentiability and polynomial-growth conditions on normalized derivatives of the initial density, we combine complementary estimates from the Gaussian transition kernel and the initial density to obtain explicit Sobolev bounds that are uniform in time on each fixed ball. Combining these bounds with existing shallow-network approximation theory gives explicit control of the approximation error and a sufficient path-norm budget. We then use truncation and localized Rademacher complexity to establish high-probability risk bounds for path-norm-constrained empirical DSM minimizers with unbounded inputs and labels. Under additional bounded-density and sub-Gaussian assumptions, suitable choices of network width and path-norm budget yield squared $L^2(P_t)$ score risk $\widetilde O(n^{-(d+3)/(2d+3)})$ at each fixed time $t>0$, where $n$ is the number of independent training samples and $P_t$ is the forward marginal. The nonasymptotic bounds retain explicit dependence on diffusion time and the regularity and tail parameters of the initial distribution. Aggregating the score errors over a time grid and applying existing sampling theory together with an early-stopping bound yields a high-probability total-variation error $\widetilde O(n^{-(d+3)/(4d+6)})$ between the generated and original data distributions under suitable sampling parameters. Here $\widetilde O$ suppresses logarithmic factors for fixed dimension, confidence level, and initial distribution.