Spectral alignment improves training and output quality of diffusion models
Revisiting Spectral Representations in Generative Diffusion Models
Machine Learning
Summary
Diffusion models create images or 3D shapes by slowly removing random noise from data. The authors found that teaching these models to also learn special patterns called spectral representations helps them train faster and produce better results. They showed that connecting spectral learning with diffusion processes reveals why this helps and developed a new method that adds a spectral alignment step during training. Their tests on images and 3D point clouds proved this approach consistently improves the quality of generated outputs.
What this means in practice
- •For computer graphics engineers: Enhance the quality of generated images and 3D models by integrating spectral alignment into diffusion model training.
- •For machine learning engineers: Speed up convergence and improve sample quality in diffusion-based generative pipelines by applying the proposed spectral regularizer.
Authors
Yuehao Wang, Peihao Wang, Hanwen Jiang, Ziyi Yang, Qixing Huang, Zhangyang Wang
Abstract
Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality. Code is released at https://github.com/yuehaowang/spectral-reg-diffusion.