Speech encoders vary in robustness to background noises and pauses
Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoders
SoundComputation and Language
Summary
Background noises that are not speech can change how speech is represented in computer models without always hurting task accuracy as much. The authors studied eight speech encoders and tested them on tasks like recognizing intent or emotion, adding different sounds throughout recordings or just during speech or pauses. They found that noises during pauses cause larger shifts in the speech representation, but noises during speech usually hurt performance more. This shows that just measuring changes in the internal speech representation may not fully predict how well tasks will perform.
What this means in practice
- •For voice assistant developers: Identify how different background noises impact speech recognition model reliability for more robust voice assistant performance.
- •For call center software teams: Improve speaker verification systems by understanding the effects of noise placement in recordings on identity checks.
Authors
Vsevolod Kovalev, Pranay Manocha
Abstract
Non-speech interference can change a speech representation without causing comparable task loss. We test eight frozen encoders on four tasks, adding non-speech sounds throughout recordings, during speech, or in pauses. Under whole-recording interference, embedding drift tracks task loss across seven sounds, with mean Spearman correlations of 0.81-0.88. Moving the same sound between speech and pauses changes this pattern. At quiet to moderate levels, pause interference produces larger drift, while speech interference usually causes greater loss on intent recognition, speaker verification and speech recognition. Emotion recognition shows a weaker placement effect. Pause interference also changes speech-frame representations beyond the injected region. Even below the estimated recording background, interference can change embeddings as much as repeated speech takes do. Drift helps rank the effects of different sounds, but larger drift does not consistently indicate greater task loss.