Speech quality checks can be fooled more easily than thought
How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys
Machine Learning
Summary
Speech quality metrics, which measure how good speech sounds after processing, can be tricked by small changes in the input. The paper shows that previous ways to test how easy it is to fool these metrics had mistakes. The authors propose a better testing method that reveals big differences in how vulnerable popular speech quality metrics are to attacks. They also explore attempts to fix these weaknesses and find that fixes only work against specific types of attacks and may reduce performance in other cases.
What this means in practice
- •For speech software developers: Identify how vulnerable their speech quality evaluation tools are to manipulation before deployment.
- •For voice assistant engineers: Improve robustness of speech quality metrics used as feedback to enhance voice-driven AI systems.
Authors
Ali Alavi, Donald S. Williamson
Abstract
Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack. Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four (0.31 to 0.08 for one defence). Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several. Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%. We then audit a closed attack-detect-patch loop. It hardens the predictor only in its own attack space, by less than the spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out of domain. Enhancers post-trained against patched predictors hack them far less (PESQ -0.03 versus -0.23). Code, preregistration and run outputs are released.