Privacy loss during diffusion model sampling depends on method
Privacy Accounting for Empirical Denoisers
Machine LearningCryptography and Security
Summary
Diffusion models are popular tools that create images, but they can remember details from their training data, raising privacy concerns. The paper investigates how much private information is revealed during the image generation process itself, not just during training. The authors build a safe way to measure privacy loss step-by-step by using a special, simpler version that can be analyzed exactly. They find privacy loss increases sharply at key moments when important parts of the image form, suggesting new ways to mix private and public methods for better privacy. Their tests on common datasets show that privacy depends on how the model is used at generation time, not just how it was trained.
What this means in practice
- •For machine learning engineers: Estimate privacy loss during diffusion model sampling to design safer data generation methods.
- •For privacy compliance teams: Assess privacy risks of AI-generated content based on sampling choices and stopping criteria in deployed systems.
Authors
William Cheng, Mijung Park, Dongjun Kim
Abstract
Diffusion models have emerged as the foundation of modern generative systems, yet their high memorization capacity raises privacy concerns. While differentially private (DP) training provides formal guarantees, it comes at a cost in sample quality. In this work, we study a complementary question: how much privacy loss is incurred during the sampling process itself? Directly analyzing modern neural denoisers is challenging because their sensitivity to a single training example is generally intractable. To overcome this challenge, we construct and analyze a clipped privacy-accountable empirical denoiser, a training-free estimator of the Bayes-optimal denoiser, whose per-step sensitivity can be computed explicitly, allowing each denoising step to be a Gaussian mechanism. This perspective enables privacy accounting under Gaussian DP when each training example contributes to each denoising step, and under $f$-DP trade-off functions when each generated sample uses a single randomly subsampled block of the training data. We report every $(\varepsilon, δ)$ guarantee as a certified upper bound from a privacy loss distribution accountant, and use a cumulant-based Edgeworth approximation as a fast diagnostic, characterizing where it is reliable and where it becomes anti-conservative. We also analyze the privacy loss along the denoising trajectory and show that it grows steeply near the critical window in which semantic content is formed. This observation motivates hybrid samplers that use the private empirical denoiser through the formative steps where it remains a faithful proxy for the neural denoiser, then switch to a public neural denoiser for the remaining refinement. Our results on CIFAR-10 and CelebA-64 show that the privacy of diffusion models depends not only on the training procedure, but also on sampling-time design choices such as subsampling, stopping time, and hybrid denoising.