Revisiting the Provable-Auditable Privacy Gap of DP-SGD
Machine Learning
Summary
The authors discuss how differential privacy (DP) normally gives a theoretical upper limit on how much an algorithm’s output can change if the training data changes. However, existing analyses might be too loose or pessimistic. They focus on the idea of 'empirical privacy,' which measures privacy by testing the algorithm against real attacks. The authors propose a new defense method that improves this empirical privacy without sacrificing the theoretical privacy guarantees, and they show it works well across different tests, models, and datasets.
Authors
Saloni Modi, Srivi Balaji, Yusong Zhu, Gautam Kamath, Kevin Tian
Abstract
Differential privacy (DP) has traditionally been used to provide theoretical upper bounds on an algorithm's stability to changing its training data. In modern private machine learning applications, achieving strong tradeoffs between utility and theoretical privacy is challenging, and thus one may optimistically hope that existing theoretical privacy analyses are loose. Recent work on privacy auditing has adopted a dual viewpoint, instead lower bounding the true privacy of an algorithm by constructing empirical distinguishing events. The auditing literature has thus far yielded a pessimistic outlook on the looseness of theoretical privacy bounds for DP-SGD, the de facto private training method in modern ML, as nearly-matching empirical lower bounds have been achieved under various threat models [NHSBTJCT23, AC24, CBP25]. In this work, we propose the empirical privacy lower bound of an algorithm as a concrete metric to optimize for, complementary to the theoretical upper bound. We give a lightweight defense framework that generically augments optimization methods in the ML pipeline to have significantly-improved empirical privacy on standard benchmarks. Moreover, we show that our framework comes at no theoretical privacy cost when augmenting DP-SGD, unlike previously-proposed defenses against membership inference attacks. We evaluate our defense against a broad range of audit constructions, models, and datasets to demonstrate its flexibility.