Diffusion transformers denoise images using adaptive patch-based rules
How Do DiTs Denoise? An Aggregator-Predictor View of Attention and MLPs
Computer Vision and Pattern Recognition
Summary
This paper investigates how diffusion transformers clean up noisy images. It shows that these models work by looking at small image patches and predicting clean versions based on their surrounding context. The authors explain that attention helps the model decide which parts of the image to focus on, while other parts generate the actual predictions. Their experiments with real and synthetic images support this understanding and show how their findings can improve model design.
What this means in practice
- •For computer vision engineers: Improve image denoising models by tuning component interactions based on the proposed patch-based denoising roles of attention and MLPs.
- •For machine learning model builders: Design new architectures for image generation that leverage adaptive patch predictors informed by this patchwise denoising framework.
Authors
Gyeonghun Kang, William Xingxu Chen, Xiang Cheng
Abstract
How do diffusion transformers (DiTs) represent image content and learn rules for reusing it to generalize? We develop an aggregator--predictor view of DiTs as input-adaptive local denoisers, in which context aggregation and patch prediction provide functional roles for interpreting attention and MLPs. We first establish that, for Gaussian Markov random field (GMRF) priors, the global Bayes denoiser can be approximated using only a subset of context patches, with error decaying exponentially in context radius. We further show that patchwise linear denoisers, often invoked to explain diffusion generalization, recover the empirical local denoiser with sufficiently informative aggregation. Within this view, AdaLN-modulated attention naturally parameterizes an input- and noise-dependent aggregator, while the MLP admits a learned key--value representation for tokenwise patch prediction. Our experiments support this view. DiTs trained on real images exhibit noise-dependent denoising contexts resembling those in the analytic GMRF setting, while attention mass exhibits a qualitatively corresponding broadening pattern. In a controlled CelebA64 study, shared-memory patch predictors used in place of DiT MLPs achieve competitive denoising and generative performance. Finally, cross-dataset component transplantation reveals compatibility patterns consistent with the proposed roles of QK in spatial weighting, VO in feature transport, and MLPs in patch prediction. Together, these results provide a component-level account of DiT denoising and motivate architecture-specific fine-tuning and controlled intervention.