Private decentralized learning improves accuracy with noise reduction and bias correction
Private Decentralized Optimization with Noise Reduction and Bias Correction
Machine Learning
Summary
Decentralized learning lets many devices work together to train a model while keeping their own data private, but noisy measurements and different data on each device can cause problems. The authors propose a method called Private Recursive Decentralized Optimization (PRDO) that reduces the errors caused by noise and fixes biases from uneven data. They show mathematically that their approach converges under broad conditions and that it can be more precise in protecting privacy. Their experiments demonstrate better accuracy than existing methods.
What this means in practice
- •For federated learning engineers: Improve privacy and accuracy in distributed machine learning systems handling diverse user data by reducing noise and correcting bias during optimization.
- •For internet of things developers: Build decentralized AI systems on IoT devices that maintain data privacy with less noise interference and better model convergence under varied data conditions.
Authors
Yizhao Fan, Wenjian Luo, Jiaojiao Zhang
Abstract
Private decentralized learning is affected by sampling noise, privacy noise, and decentralized bias under heterogeneous data. We propose Private Recursive Decentralized Optimization (PRDO). PRDO uses recursive estimation with same-batch gradient differences to reduce estimation errors caused by sampling and privacy noise, while its Exact Diffusion component corrects decentralized bias arising from data heterogeneity. Our analysis establishes a nonconvex convergence bound without assuming uniformly bounded data heterogeneity across nodes. It further gives a sufficient condition under which recursive gradient differences yield strictly lower query sensitivity than private Exact Diffusion, together with an example that rigorously satisfies this condition. Experiments show improved accuracy over the evaluated baselines.