AI summaryⓘ
The authors study how to make a pretrained image-generating diffusion model also good at recognizing and learning from images without losing its ability to create new ones. They treat noisy versions of the same image during the model's generation process as different views, using this idea to train the model to better understand image features via contrastive learning alongside its usual task. By making small updates to the model (using LoRA) and keeping most of it frozen, their method D³CL improves the model's performance on image recognition tasks while still creating high-quality images. They also explore which parts of the generation process are most useful for learning representations. Overall, the authors show a way to efficiently adapt these powerful models for both generating and understanding images.
Text-to-image diffusion modelsDenoising dynamicsContrastive learningRepresentation learningLoRA (Low-Rank Adaptation)Stable DiffusionLinear probingFID (Fréchet Inception Distance)Unconditional generationParameter-efficient updates
Authors
Yasong Dai, Zeeshan Hayder, David Ahmedt-Aristizabal, Hongdong Li
Abstract
Text-to-image diffusion models exhibit unprecedented generative capability and contain rich intermediate representations that can be useful for discriminative vision tasks. Motivated by this observation, we study a focused question: how can the denoising dynamics of a pretrained diffusion model be adapted to support discriminative representation learning while preserving its generative behavior under parameter-efficient updates? We present D$^3$CL as an investigation of this question. Our key observation is that noisy latents at different diffusion timesteps can be interpreted as stochastic views of the same underlying image, enabling a contrastive objective to be coupled with the standard denoising reconstruction loss. This formulation provides a simple way to probe the interaction between generative denoising and discriminative representation learning without training from scratch. To keep the adaptation lightweight, we apply LoRA updates to a pretrained Stable Diffusion backbone while freezing the original model parameters. D$^3$CL provides strong empirical evidence that reconstruction and noise-level contrastive objectives can be complementary: on ImageNet-1K, it obtains 80.1% linear-probing accuracy and an FID of 5.56 for $256 \times 256$ unconditional generation. Additional ablations on the design space suggest that the usefulness of diffusion features depends on where and how denoising states are sampled. These results establish D$^3$CL as a parameter-efficient adaptation framework for pretrained diffusion models, showing that noise-level contrastive learning can structure denoising representations for discriminative tasks while maintaining generative performance.