Audio model maps synthetic speech flaws to phoneme times
XSQ-AST: An Explainable Audio Spectrogram Transformer Framework for Localising Synthetic Speech Artifacts
Machine LearningSound
Summary
It is hard to know exactly when synthetic speech sounds strange because most tests give only overall scores. The authors created XSQ-AST, which uses a speech quality model combined with tools that match sounds to words and ways to highlight important parts in the audio to find where problems happen. They tested it with 40 people who listened and marked issues, finding that their approach matched what listeners heard. Different methods worked better for different kinds of speech flaws, and the system could tell real issues from random noise better than guessing.
What this means in practice
- •For speech technology developers: Pinpoint synthetic speech errors in time to improve text-to-speech systems without retraining models.
- •For quality assurance teams: Use temporal artifact localization to better identify and diagnose problematic audio segments during speech product evaluations.
Authors
Ben Heritage, Luca Resti, Mónica Villanueva Aylagas, Timothy Mehlenbacher, Konrad Tollmar, James Alfred Walker
Abstract
Localising artifacts in synthetic speech remains challenging, as most evaluation methods yield only global quality scores. This paper presents XSQ-AST, a framework that combines the SQ-AST speech quality model with WhisperX phoneme alignment and multiple saliency methods to produce temporally localised artifact diagnostics without model retraining. Saliency maps are projected onto continuous distributions via kernel density estimation and onto phoneme boundaries via phoneme-discretised saliency maps. A 40-participant listening test validated the framework across five perceptual dimensions. Attention Rollout, Attention Flow and an adapted GradCAM produced temporal distributions that correlated with listener highlights, with different methods best suited to different artifact types. An AUC-ROC analysis confirmed discrimination above chance.