Differential privacy method improves federated speech model training accuracy
Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs
Computation and Language
Summary
Training large speech models with user data needs to keep privacy protected. Existing privacy techniques that treat all parts of the model equally do not work well because speech models have very different behaviors between the sound-processing part and the language part. The authors found that splitting privacy protections into two separate groups for these parts keeps accuracy high and still offers strong privacy guarantees. Their method reduces errors in speech recognition while protecting against attacks that try to recover a person's voice from training data.
What this means in practice
- •For voice assistant developers: Improve privacy protection in federated learning while maintaining speech recognition accuracy for multilingual voice assistants.
- •For mobile device manufacturers: Enhance on-device speech model training with stronger privacy guarantees against voice inversion attacks and balanced noise addition.
Authors
Jordi Luque, Fernando López, Aleix Sant
Abstract
Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer \emph{cross-component budget collapse}, dragging word error rate (WER) far from flat global clipping or collapsing training entirely. When the norm imbalance is milder, adaptive single-pool methods partially recover, confirming that collapse severity scales with the inter-component norm ratio. We empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures. We then propose \emph{$α$-split}, a two-pool allocation that normalises encoder and LLM parameters into independent pools, and show that joint $\ell_2$ sensitivity and the original $(\varepsilon,δ)$-DP guarantee are unchanged. At architecture-calibrated $α$, our method recovers WER utility compared to flat DP, while granting the encoder $4.47{\times}$ tighter per-component noise protection against speaker voice-based gradient-inversion attacks at only $+2.6\%$ LLM noise overhead.