Unveiling the Secret of AdaLN-Zero in Diffusion Transformer
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied a part of the Diffusion Transformer (DiT) model called adaLN-Zero, which helps the model generate images better than a similar method called adaLN. They found that the key to adaLN-Zero's success is a special way of starting the model's settings, called zero-initialization. Based on this, they created a new initialization method called adaLN-Gaussian that makes training more efficient. They also developed an improved version named SE-adaLN-Zero, inspired by a known structure in neural networks. Their tests on various datasets, including ImageNet1K, show these new methods work well not just for creating images from labels but also from text descriptions.
Diffusion Transformer (DiT)adaLN-Zerozero-initializationSE-like structureinitialization strategyImageNet1Kconditioning mechanismoptimization efficiencytext-to-image generation
Authors
Jie Zhu, Mingyu Ding, Boqiang Duan, Leye Wang, Jingdong Wang
Abstract
Diffusion transformer (DiT), a rapidly emerging architecture for image generation, has gained much attention. However, despite ongoing efforts to improve its performance, the understanding of DiT remains superficial. In this work, we delve into and investigate a critical conditioning mechanism within DiT, adaLN-Zero, which achieves superior performance compared to adaLN. Our work studies three potential elements driving this performance, including an SE-like structure, zero-initialization, and a "gradual" update order, among which zero-initialization is proved to be the most influential. Building on this understanding, we propose an analysis-guided initialization strategy, termed adaLN-Gaussian, which serves both as an empirical validation of our analysis and as a practical initialization method that consistently improves optimization efficiency. On the other hand, inspired by the SE-like structure, we introduce an improved conditioning mechanism called SE-adaLN-Zero. Extensive experiments following DiT on four datasets, especially on ImageNet1K demonstrate the effectiveness and generalization of adaLN-Gaussian and SE-adaLN-Zero. Beyond class-to-image generation, we also evaluate the generalization of the two improved methods on text-to-image generation.