AccentCL improves English accent detection and adds new accents easily
AccentCL: Robust Accent Classification with Incremental Expansion
Computation and LanguageMachine Learning
Summary
AccentCL helps computers recognize different English accents better by dealing with unbalanced data and changes in how speech is recorded. The system starts with a base model and then can learn to identify new accents without forgetting the old ones. It uses special techniques to keep knowledge from previous tasks while adding new categories, all without retraining from scratch. Tests showed AccentCL keeps good accuracy on old accents and learns new accents reasonably well.
What this means in practice
- •For speech technology developers: Create speech recognition systems that adapt to new English accents over time without retraining fully.$Commercial implications: Enables development of commercial speech services that expand accent coverage flexibly and maintain accuracy in diverse accents.
- •For call center software engineers: Integrate incremental accent recognition to improve automated call routing and customer interaction analysis.
Authors
Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah, Ricardo Gutierrez-Osuna
Abstract
Accent classifiers are typically trained with a fixed label inventory and cannot accommodate new accent categories as new data becomes available. Moreover, accented speech corpora often exhibit substantial class imbalance and/or domain shift due to differences in recording conditions across corpora. We present AccentCL, a class-incremental learning framework for English accent classification that is robust to class imbalance and cross-corpus domain shift. AccentCL extracts multi-layer representations from a frozen Whisper-Large-v3 encoder, optimized with an imbalance-aware cross-entropy loss to reduce bias toward the majority accent classes and a domain mean alignment loss that minimizes distributional mean shift across training corpora. The label space is then expanded via replay-based continual learning, using the frozen base model for knowledge retention and an old-to-new margin loss to reduce overprediction on newly added classes. On a five-class accent classification task, AccentCL achieves 77.1% balanced accuracy and a 76.9% macro-averaged F1 score. We further evaluate the model's ability to incrementally incorporate two new accent categories: Spanish-accented and Chinese-accented English. When adding Spanish-accented English to the pretrained model, AccentCL attains an F1 of 83.3% on the new class while retaining 77.3% balanced accuracy on the base classes. When subsequently adding Chinese-accented English, it achieves 61.8% F1 on the new class while preserving 77.6% balanced accuracy on the previously learned classes. These results show that AccentCL enables robust regional accent classification while allowing new accent categories to be added without full retraining.