Differentially private learning rates transfer across model sizes
Towards Hyperparameter Transfer for Differentially Private Optimization
Machine LearningCryptography and Security
Summary
Training large AI models while protecting data privacy is tricky because added noise and special limits on learning can mess up usual tuning methods. The authors explain why a popular tuning method called Maximal Update Parametrization ($μ$P) doesn’t work well with privacy rules. They develop DP-$μ$P, a new way to adjust learning rates that accounts for privacy constraints, making it possible to reuse tune settings for different model sizes without extra work. Their tests show this approach works well on common model types, saving time and computing effort.
What this means in practice
- •For machine learning engineers: Reuse hyperparameter settings across different model sizes when training private models, reducing tuning time and computational costs.
- •For mobile app developers: Deploy privacy-preserving deep learning models with efficient training steps tuned across architectures, aiding feature learning under data privacy rules.
Authors
Tianze Wang, Zhiqi Bu, Linjun Zhang
Abstract
Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization ($μ$P) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP-$μ$P, a theoretically grounded framework that extends $μ$P to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP-$μ$P enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance of individually tuned baselines while significantly reducing computational costs.