Accent classification works better when tested on different speakers and datasets
Revisiting Accent Classification with DistilHuBERT: Speaker-Disjoint Evaluation and Cross-Corpus Generalization
Computation and LanguageMachine LearningSound
Summary
Many accent classification systems show they work well when tested on data with the same speakers as the training data, but this doesn't reflect real-life use. The authors show that when testing on speakers not in the training data and on different datasets, the performance drops significantly. They also find that training on diverse external datasets can help but still does not fully solve the problem. Their work suggests that future tests should include speaker-disjoint and cross-dataset evaluations to better understand how the models perform in real-world situations.
What this means in practice
- •For voice assistant developers: Improve accent recognition systems by evaluating models on speakers and datasets not seen during training to better handle real users.
- •For call center technology teams: Create more reliable accent detection for speaker verification or routing by training on balanced external accent data and testing for generalization.
Authors
Xu He, Homayoon Beigi
Abstract
Recent accent classification systems can achieve high accuracy in standard in-corpus evaluations. However, when speakers overlap between the training and test sets, such results may overestimate model robustness compared with real-world deployment scenarios. This study revisits the robustness of DistilHuBERT-based accent classification by using L2-ARCTIC and multiple external English accent datasets. The comparison covers file-level and speaker-disjoint splits, along with balanced external-data training and teacher-student pseudo-labeling. The large performance gap between in-corpus and external evaluations suggests that file-level evaluation substantially overestimates performance, while speaker-disjoint evaluation can provide a more realistic estimate of the generalization to unseen speakers. In this study, models trained on external data alone do not reliably generalize to L2-ARCTIC. Among the tested configurations, balanced external-data training offers the most promising supervised setting. The pseudo-labeling experiments further indicate that confidence-based filtering alone is insufficient under domain mismatch. Overall, this study suggests that robust accent classification evaluation should simultaneously include speaker-disjoint evaluation and cross-corpus testing.