Deep Learning under Fractional-Order Differential Privacy
2026-05-11 • Cryptography and Security
Cryptography and SecurityMachine Learning
AI summaryⓘ
The authors introduce FO-DP-SGD, a new version of a privacy-focused training method called DP-SGD. Unlike the traditional method that uses only the current gradient, their approach combines current and past gradients in a weighted way, adding a kind of memory to the learning process while keeping privacy protections intact. By controlling how much of the current information is used, their method maintains privacy guarantees using existing mathematical tools. Their experiments show that FO-DP-SGD improves accuracy and privacy compared to other private training methods on popular image datasets.
Differentially Private SGDStochastic Gradient DescentRényi Differential PrivacyPoisson SubsamplingGradient ClippingGaussian NoisePrivacy AccountingFractional MemoryMachine Learning PrivacyOptimization Algorithms
Authors
Mohammad Partohaghighi, Roummel Marcia
Abstract
Differentially private stochastic gradient descent (DP-SGD) is a standard approach to privacy-preserving learning based on per-example clipping, subsampling, Gaussian perturbation, and privacy accounting. Classical DP-SGD releases a noisy version of the current clipped subsampled gradient sum. We propose Fractional-Order Differentially Private Stochastic Gradient Descent (\textbf{FO-DP-SGD}), a mechanism-level extension that replaces this current-only query, before Gaussian noise is added, with a fractional recursive query combining the current clipped sum with a finite-window, power-law-weighted aggregation of previously released private sum-level outputs. This injects fractional memory into the release mechanism while preserving the standard \emph{sum-then-noise-then-divide} structure. Under add/remove adjacency with Poisson subsampling, the current-step sensitivity analysis shows that the only newly data-dependent term is the scaled current clipped sum. Hence, conditioned on the private history, the effective \(\ell_2\)-sensitivity is at most \(βC\), where \(C\) is the clipping threshold and \(β\in(0,1]\) controls the current-step contribution. Thus, FO-DP-SGD admits standard per-step Rényi differential privacy accounting via a Poisson-subsampled Gaussian mechanism with effective noise-to-sensitivity ratio \(σ/β\), and composes to yield overall \((\varepsilon,δ)\)-differential privacy guarantees. FO-DP-SGD provides a framework for studying long-memory effects in private optimization. The fractional order, memory window, and mixing coefficient govern the trade-off among current-step sensitivity, signal retention, and private-history influence. Experiments on SVHN, CIFAR-10, and CIFAR-100 show improved test accuracy and privacy--utility performance over DP-SGD and private baselines including DP-Adam, DP-IS, SA-DP-SGD, ADP-AdamW, DP-SAT, and DP-Adam-AC.