Information theory bounds how well private models reveal training data
From Pixels to Privacy-Sensitive Features: An Information-Theoretic Framework for Reconstruction Bounds
Machine Learning
Summary
When machine learning models are trained to protect privacy, it's not clear exactly what an attacker might still learn about the original data. This paper develops a mathematical way to estimate the minimum error any attacker would have when trying to reconstruct training data from such a model. The method measures how much information about a data record the training reveals, and combines it with how uncertain the data was to start with. The authors show this approach works on image data and gives meaningful limits on reconstruction even when strong attacks try to get private details back.
What this means in practice
- •For machine learning engineers: Assess expected minimum reconstruction error from privacy-preserving training setups to choose privacy parameters more reliably.
- •For privacy compliance teams: Quantify unavoidable privacy risks in models trained with differential privacy methods to support risk management.
Tested on simulated data.
Authors
Henry Bourne, Robert F. Allison
Abstract
What can an adversary recover from a model trained with DP-SGD? Its $(\varepsilon,δ)$ guarantee does not answer this question directly. We derive a distortion bound that can be computed before training and applies even to an adversary who observes the parameter trajectory during training, knows every other training record and the population distribution from which the training records are drawn. The analyst may choose the representation in which error is measured, allowing the bound to target the aspects of reconstruction that are privacy-sensitive in a given application. Our information-theoretic framework separates two contributions: the information about a record revealed by training, measured through conditional mutual information, and the uncertainty already present in the data distribution, measured through source entropy. Rate-distortion theory combines them into a lower bound on the expected error of every reconstruction attack. For DP-SGD, the leakage bound depends only on the sampling rates and noise multipliers, while the source entropy is lower-bounded from a sample with finite-sample confidence. An orthogonal block construction keeps the resulting bound informative for high-dimensional images. We instantiate the framework in pixel space and in LPIPS feature space, obtaining, to our knowledge, the first lower bound on reconstruction distortion measured directly by LPIPS. On a range of image datasets the bounds establish non-trivial unavoidable expected reconstruction error within heuristically recommended privacy-budget ranges and beyond. Experiments compare the bounds with strong reconstruction attacks and identify the principal sources of the remaining slack.