Curvature aware clipping improves privacy preserving neural network training
Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information
Machine LearningCryptography and Security
Summary
Training neural networks with privacy protection often requires limiting how big gradients can get, but this causes biases that hurt the learning process. The authors propose a new method that uses an approximate understanding of the shape or curvature of the training landscape to decide how to clip gradients more effectively. This approach also adds noise in a way that respects the privacy guarantees. Their experiments show that this method works better than previous ways to reduce clipping bias while preserving privacy.
What this means in practice
- •For machine learning engineers: Train deep learning models on sensitive data with less accuracy loss by using curvature-aware clipping to reduce gradient bias.
- •For data privacy teams: Implement improved differentially private training methods that better balance model utility and privacy by leveraging approximate curvature information.
Authors
Xincheng Xu, Thilina Ranbaduge, Thierry Rakotoarivelo, David Smith
Abstract
Differentially Private Stochastic Gradient Descent (DPSGD) has become the de facto standard for training deep neural networks with rigorous privacy guarantees. However, the utility of DPSGD is constrained by gradient clipping bias and DP noise. While existing work has tried to address these issues, the geometric misalignment caused by clipping bias remains a critical challenge. To bridge this gap, we propose a novel framework, Curvature-Aware Differentially Private Stochastic Gradient Descent (CA-DPSGD), which leverages approximate second-order information to construct an optimal, curvature-aware clipping space. CA-DPSGD integrates a rectified estimator for the curvature matrix and is coupled with an anisotropic noise mechanism to ensure privacy guarantees. Extensive empirical experiments demonstrate that our approach outperforms state-of-the-art clipping methods.