Fair Finetuning Mitigates Distribution Inference Attacks
2026-06-01 • Machine Learning
Machine LearningArtificial IntelligenceCryptography and Security
AI summaryⓘ
The authors study how machine learning models can accidentally reveal sensitive group information about the data they were trained on, even if someone only has limited access to the model. They introduce a method called Fair Fine-tuning (FFt) that adjusts a model using data from a complementary group while enforcing a fairness rule called Equalized Odds (EO). Their work mathematically shows how reducing fairness disparities can also limit privacy risks from these leaks. Testing FFt on different types of data, they show it effectively reduces attackers' ability to determine group proportions. This connects fairness techniques with protecting privacy in a new, formal way.
Distribution Inference AttackEqualized OddsFair Fine-tuningAdversarial AdvantageDifferential PrivacyProperty UnlearningSensitive AttributesMachine Learning FairnessPopulation-level PrivacyBlack-box Access
Authors
Rakshit Naidu
Abstract
Machine learning models trained on sensitive data can inadvertently leak population-level information about their training distributions -- a threat known as distribution inference attack (DIA). An adversary with black-box access can infer sensitive demographic properties, such as subgroup proportions, without observing any training data directly. While defenses such as differential privacy and property unlearning have been proposed, the link between fairness constraints and distributional leakage remains unexplored. We propose Fair Fine-tuning (FFt): a trained model is fine-tuned on samples from the complementary distribution under an Equalized Odds (EO) constraint. We provide a complete theoretical characterization, proving the tight bound $\text{Adv}(\mathcal{A},M_f) \le Δ_{\text{EO}} \cdot W$, where $W$ quantifies how distinguishable the two training distributions are by their sensitive-attribute composition. We also establish a necessary condition for FFt to reduce adversarial advantage and prove tightness of the bound. We evaluate across six datasets spanning tabular (ACS Income, COMPAS, German Credit), image (UTKFaces), and NLP (Bias in Bios) modalities. Rehearsal-based FFt consistently reduces the adversarial accuracy gap below the detection threshold $τ!=!0.1$ across all settings; on ACS Income, the gap falls from $\sim!15%$ to under $4%$. Our work provides the first formal bound connecting a model's measured EO disparity directly to its adversarial advantage in the DIA game, opening a new avenue for unified fairness-and-privacy defenses.