Speech models gain robustness without retraining each task
Transferable Adversarial Robustness for Speech Foundation Models via Hierarchical Stabilization
Machine Learning
Summary
Many speech recognition systems use a part called a backbone that stays the same while different smaller parts learn specific tasks. Making these systems tough against tricky inputs (called adversarial attacks) usually means changing the backbone, which is slow. The authors found a way to improve the backbone’s resistance to attacks ahead of time by stabilizing how it processes sounds inside, not just at the end. This means new tasks can stay fast to learn while still being more secure against attacks. They showed this works well on popular speech models across several tasks.
What this means in practice
- •For speech recognition developers: Build speech systems that resist adversarial attacks without costly retraining of core models for each new task.
- •For voice assistant engineers: Enhance user voice commands' security by improving the underlying speech models' robustness before deployment.
Authors
Aref Mousavi, Shahab Sherafat, Kiarash Kiani Feriz, Amirparsa Safari, Raoof Zare Moayedi, Mohammad Hossein Rohban, Mohammad Sabokrou
Abstract
Frozen speech foundation models (SFMs) make downstream adaptation efficient: the backbone can stay fixed while a task learns layer fusion and a lightweight classifier. Full adversarial fine-tuning is a standard route to robustness, but generating adversarial examples and updating the backbone for every task sacrifices that efficiency. We ask whether robustness can instead be learned before future tasks are known. For a frozen backbone and linear classifier, robustness can be understood through the interaction between representation stability and decision-boundary margin. This leads directly to our design: we stabilize representations across the hidden layers, rather than only the final layer, while preserving clean representations; after clean adaptation selects the layer mixture, we keep it fixed and enlarge only the classifier margin, without downstream adversarial examples. We evaluate Wav2Vec2, HuBERT, and WavLM Large on four tasks under adaptive 30 dB attacks. Across 12 backbone-task pairs, hierarchical robustification improves robust accuracy by 46.4 pp, while margin refinement adds 4.0 pp for 1.1 pp of clean accuracy. Code and configurations are available at https://github.com/arefmousavi/hierarchical-robust-sfm.