Psychoacoustic noise injection defends real-time speech dialogue models
Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models
SoundArtificial IntelligenceComputation and LanguageHuman-Computer Interaction
Summary
Speech-to-speech dialogue systems that listen and talk at the same time can be tricked by tiny sounds an attacker adds without people noticing. The authors show how attacks can hijack, mute, or confuse these systems with subtle noise under human hearing thresholds. They create a technique called psychoacoustically aligned latent smoothing that adds special noise inside the model training to make it robust. This approach greatly reduces attacks while keeping the speech clear and costs nothing extra when used for listening and talking.
What this means in practice
- •For voice assistant developers: Protect full-duplex voice assistants from subtle, imperceptible voice command attacks by integrating psychoacoustically aligned noise injection during training.
- •For telecommunication security teams: Enhance security of real-time speech communication systems by reducing adversarial hijacking and mute attacks that work under psychoacoustic limits.
Authors
Kian Shamsaie, Iman Modarressi
Abstract
End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback--Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds.