Certified defenses for vision transformers improved with new privacy bounds
Revisiting Certified Defense with Differential Privacy on Vision Transformers
Cryptography and Security
Summary
Giving computer vision models strong guarantees against attacks is important, especially for new architectures like Vision Transformers. The authors found that existing privacy-based defenses that worked well for older models fail when applied directly to Vision Transformers because of how sensitivities (measures of changes) are calculated. By changing the mathematical constraints and developing new privacy guarantees, they fixed these failures and improved protection without losing accuracy. This work helps build more secure Vision Transformer models using differential privacy techniques.
What this means in practice
- •For computer vision engineers: Deploy more robust Vision Transformer models with mathematically proven defenses against image perturbations using enhanced differential privacy methods.
- •For security software developers: Integrate improved privacy-certified mechanisms into AI systems to better guarantee resistance to adversarial attacks on visual data.
Authors
Jun Yan, Weiquan Huang, Qixian Zhang, Yan Bai, Shutai Zhang
Abstract
Certified defenses that incorporate differential privacy have proven effective on Convolutional Neural Networks (CNNs), furnishing rigorous robustness guarantees against norm-bounded adversaries. However, the certified robustness behavior of Pixel Differential Privacy (PixelDP) remains largely unexplored with the self-attention architecture now dominating the deep-learning landscape. Given that the Transformer has a profound impact on our daily applications from the digital world to the physical world, it is crucial to study certified robustness through differential-privacy-style stability. To fill this research gap, we revisit this construction in Vision Transformers and identify a failure mode that is largely hidden in the convolutional setting. When noise is injected after the patch embedding, the Laplace mechanism with the inherited grouped $\ell_1$ sensitivity bound collapses to chance-level accuracy across noise scales, whereas the Gaussian mechanism remains trainable. This contrast isolates the source of failure: not the injected noise itself, but the geometry of the sensitivity constraint. We show that the attenuation induced by the inherited $Δ_{1,1}$ projection increases with layer width and kernel size according to a random-matrix scale $C/(\sqrt{M}+\sqrt{N})$. Replacing the $\ell_1$-type constraint with a spectral-norm constraint eliminates the collapse across datasets and architectures, but creates a fundamental obstacle: the repaired models no longer satisfy the sensitivity condition required by the standard Laplace certificate. We resolve this mismatch by deriving a dimension-free $(\varepsilon,\ δ)$-privacy guarantee for the Laplace mechanism under $\ell_2$ sensitivity through concentration of the privacy loss.