Better batches lead to improved privacy and accuracy in machine learning
Better Batches, Better Privacy: Random Allocation for DP-BandMF
Cryptography and Security
Summary
Training machine learning models while keeping data private often requires adding noise and carefully choosing batches of data. The authors studied and improved a method called DP-BandMF by using a new way to select batches called random allocation, which made the training more accurate. They also made the privacy calculations much faster by using a special GPU design. Finally, they showed that choosing batch selection and noise strategies together rather than separately can further reduce errors.
What this means in practice
- •For data scientists: Improve accuracy when training machine learning models that must protect sensitive data by adopting better batch selection and noise strategies.
- •For ai infrastructure engineers: Speed up privacy cost accounting for differentially private training using optimized GPU implementations to reduce computational overhead.
Authors
Ryan McKenna, Arun Ganesh
Abstract
The current state of the art for batch selection and noise addition in differentially private training is a combination of DP-BandMF (informally, DP-SGD where the noise added in two different steps can be correlated if the steps are within $b$ of each other) and privacy amplification by sampling via Monte Carlo accounting. First, whereas prior batch selection strategies for DP-BandMF were inspired by Poisson sampling, random allocation is known to outperform Poisson sampling for DP-SGD. We therefore propose and analyze new batch selection strategies for DP-BandMF inspired by random allocation, improving RMSE by up to $1.85\times$ over previous work. Second, Monte Carlo accounting typically requires drawing $Ω(1/δ)$ samples from a privacy loss distribution, making accounting costly. We increase sampling and scoring throughput by two orders of magnitude through a carefully optimized GPU implementation. Finally, previous work chose the DP-BandMF strategy matrix and the batch selection strategy independently. We give a simple framework for choosing these parameters jointly, reducing RMSE by up to $13.4\%$ over fixed strategy matrices.