Speech rhythm assessment improves using amplitude envelope features
Automated Assessment of L2 Speech Rhythm Using Low-Frequency Amplitude Modulations
Sound
Summary
Assessing how well people speak a new language includes checking their speech rhythm, but traditional methods need a detailed breakdown of speech sounds, which can be messy when people make mistakes. The authors propose a new way that listens to the overall loudness changes in speech instead of focusing on individual sounds. This method uses a special kind of artificial intelligence to measure speech rhythm and matches human judgments better, especially for less fluent speakers. Their approach avoids errors from aligning speech sounds and works directly from the sound wave patterns.
What this means in practice
- •For language learning platforms: Integrate rhythm evaluation directly from audio amplitude to provide more accurate feedback on learner fluency without complex speech alignment.$Commercial implications: Offers a novel assessment feature for commercial language apps improving non-native speech feedback reliability.
- •For call center trainers: Assess call agents' speech fluency and prosody more robustly, especially for agents with accents or variable speech quality.
Authors
João Lima, Lucas Ueda, Paula Costa
Abstract
Automated Speaking Assessment of non-native speech must effectively evaluate prosody, including speech rhythm, to align with human perception. However, commonly employed rhythm metrics rely on segmental duration, requiring an additional alignment step, which is error-prone in non-native speech containing disfluencies and mispronunciations. We propose an acoustics-based assessment approach that employs a convolutional neural network to extract rhythm features directly from the speech amplitude envelope, motivated by evidence linking low-frequency modulations to rhythm perception. The proposed models are trained on a proficiency score regression task using the speechocean762 dataset and compared against duration-based models. Our results show that a model using the amplitude envelope's first derivative achieves the highest correlation with human-assigned scores on the Fluency and Prosody dimensions, producing significantly lower errors than one using segment durations among less fluent speakers. The findings support acoustic envelope features as robust, alignment-free alternatives for L2 rhythm assessment. Code is released publicly.