Adaptive model attacks reveal underestimated privacy risks in image training data
Adaptive Model Inversion Attacks Generalize a Privacy-Robustness Tradeoff
Machine LearningCryptography and Security
Summary
Protecting privacy in machine learning is hard because hackers can sometimes recreate images used in training. This paper shows that current tests underestimate how much private data can be leaked, especially when attacks are adapted to the defenses in place. The authors also found that the way we measure privacy leaks changes the results, meaning some protections might seem better than they are. Importantly, they discovered a strong link between how robust a model is to adversarial attacks and how vulnerable it is to privacy leaks. This suggests that making models more robust might increase privacy risks, posing a tricky challenge for defenders.
What this means in practice
- •For machine learning engineers: Assess and improve privacy defenses in image recognition models by testing adaptive reconstruction attacks on real datasets.
- •For security analysts: Evaluate links between privacy vulnerabilities and adversarial robustness to better prioritize model security strategies.
Authors
Shailen Smith, Rasmus Torp, Adam Breuer
Abstract
In this paper, we show that standard evaluations of high-resolution Model Inversion Attacks (MIAs) significantly underestimate training-data privacy leakage. State-of-the-art privacy defenses, standard training techniques such as MixUp and Adversarial Training, and undefended models all leak training images at rates 1.16 to 6.59 times higher on FaceScrub under simple adaptive changes to the attack, with the largest increases among defenses reporting the strongest privacy. We further show that measured leakage depends on the feature basis of the external classifier used to evaluate reconstructions: for the same reconstructed images, an adversarially trained Inception evaluator identifies the targeted identity at different rates than the standard Inception evaluator. Our results suggest that standard MIA evaluation can mistake optimization and measurement failures for privacy. These underestimated leakage rates also concealed a broader relationship between privacy and adversarial robustness. Once we adapt the attack and vary the evaluator, reconstruction leakage closely tracks adversarial robustness across recent defenses and standard training regimes, suggesting that robustness provides an attack-agnostic proxy for reconstruction vulnerability that applies far more broadly than previously theorized. This raises an open question: can a practical defense reduce training-data reconstruction without paying a corresponding cost in adversarial robustness?