Spectral learning improves training and quality in diffusion models
Revisiting Spectral Representations in Generative Diffusion Models
Machine Learning
Summary
Diffusion models create images or 3D shapes by gradually removing noise through a process guided by mathematics involving randomness. The authors explore how learning certain patterns, called spectral representations, that come from comparing related and unrelated noisy data can help these models learn better. They show that aligning these spectral patterns with the diffusion process leads to easier training and better final results. Their method adds a new part to the training that encourages this alignment, improving generation quality across different types of data.
Diffusion modelsSpectral representationsSelf-supervised learningPerturbation kernelsStochastic noiseScore distillationGenerative modelsGaussian noiseRepresentation alignment3D point clouds
Authors
Yuehao Wang, Peihao Wang, Hanwen Jiang, Ziyi Yang, Qixing Huang, Zhangyang Wang
Abstract
Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality. Code is released at https://github.com/yuehaowang/spectral-reg-diffusion.