Auditing real privacy risks in federated synthetic data sharing
Canaries in the Bank: Auditing User-Level Privacy in Private Evolution
Cryptography and SecurityArtificial IntelligenceMachine Learning
Summary
Sharing data while keeping individual information private is hard. Private Evolution is a method that creates fake data to protect users’ exact details, but it’s unclear if attackers can still figure out real info. The authors tested ways to secretly insert known signals into the fake data to see how much private info leaks. They found natural language attacks leak much less than the worst-case theoretical limits, while some special random data attacks come closer to the privacy risk limit. This helps understand the gap between theoretical privacy guarantees and what can happen in practice when following the method.
What this means in practice
- •For privacy engineers: Assess actual privacy risks in federated synthetic data systems by empirically auditing protocol-compliant candidate manipulation.
- •For data security teams: Improve monitoring of data anonymization methods by detecting potential information leakage closer to theoretical privacy bounds.
Authors
Sai Aparna Aketi, Enayat Ullah, Shripad Gade
Abstract
Private Evolution (PE) generates high-fidelity synthetic data in federated settings without exposing users' raw data. It aggregates clipped user votes over a shared candidate bank into a differentially private histogram, with noise calibrated to the worst-case user contribution. However, it is unclear whether an adversary can realize this worst-case privacy loss while following the PE protocol. We introduce a protocol-aware empirical audit in which the server commits to a single shared candidate bank and replaces roughly 1% of its entries with probes derived from a known, non-private canary. We evaluate eight attacks, including an unchanged-bank baseline, exact copies, plausible paraphrases, and high-entropy synthetic nonces. Experiments on Yelp and Sentiment140 show that natural-text attacks remain substantially below the theoretical DP bound, while nonce-based attacks yield considerably stronger bounds and come closest to the mechanism's privacy ceiling. These results quantify the gap between formal worst-case privacy and leakage achievable through protocol-valid candidate-bank manipulation.