Decentralized learning protects privacy with gossip and control techniques
Privacy Preserving Gossip Learning
Machine Learning
Summary
Training machine learning models often requires sharing private data, which raises privacy concerns. The authors propose a new method where each participant only has one private data point but all share and update the model together without revealing their own data. They cleverly use ideas from tuning models without forgetting past data, add noise to keep data private, and communicate updates through a decentralized gossip approach. This ensures that private information stays protected while still allowing the model to learn effectively.
What this means in practice
- •For mobile app developers: Develop decentralized apps that train AI models on-device while protecting individual user data with privacy guarantees.
- •For internet of things engineers: Coordinate distributed sensor devices to collaboratively update models without exposing private sensor readings.
Authors
Erkan Bayram, Mohamed-Ali Belabbas, Tamer Başar
Abstract
We propose a decentralized privacy-preserving learning algorithm in which each agent holds a single private sample and a shared model. Samples are learned sequentially, and each update must preserve the endpoint mappings at previously learned samples while protecting private data. This gives each agent three roles: (i) a learner that updates the model parameters, (ii) a teacher whose sample is learned at the current iteration, and (iii) a protected agent whose sample has already been learned. We build on Tuning without Forgetting (TwF) method to preserve previously learned mappings and show that TwF provides an indistinguishability guarantee for the learner whenever the set of protected agents contains another sample with the same label. For the teacher, we formulate a minimax optimal control problem that models the differential privacy noise as a worst-case disturbance to prevent performance loss while maintaining the same level of privacy for the gradient. For the protected agents, we compute the projections locally and aggregate them using a private push-sum gossip protocol. We prove geometric convergence of the decentralized gossip algorithm and of the distributed projection for TwF.