Cross language emotion recognition works well with little data
How Language-Independent Are Emotional Attributes? A Study on Training Data Scaling and Cross-Lingual Generalization
Sound
Summary
Recognizing emotions in speech usually needs lots of examples, but such data exists mostly for a few languages like English. The authors studied how emotion recognition models trained on one language can be adapted to another low-resource language using less data. They found that for some emotions, like arousal, adapting a model works well even with just two hours of new language data. For other emotions, like valence, more data—up to 10 to 20 hours—is needed to get similar performance as training a model only on the new language.
What this means in practice
- •For voice assistant developers: Improve emotion detection for assistants in less-studied languages by adapting existing English or Mandarin models with limited local speech data.
- •For customer service software teams: Deploy emotion-aware call analytics for languages with minimal training data by fine-tuning cross-lingual models instead of retraining from scratch.
Authors
Dániel Halmai, Gábor Gosztolya
Abstract
Current state-of-the-art Speech Emotion Recognition (SER) systems rely on lots of training data, which is available only for a few languages such as English. For low-resource languages, using a cross-lingual model or adapting it to the target language might be an alternative to training a mono-lingual model from scratch. However, it is not clear what constitutes a "low-resource", i.e. where model adaptation loses its edge over the mono-lingual approach. We investigate this topic by training and adapting several SER models on English and Mandarin data, focusing on the emotional attributes of arousal, valence and dominance. Our results show that, for arousal, model adaptation is beneficial even with two hours of data, while for valence 10-20 hours of adaptation material is required to match the performance of a mono-lingual model trained with 100 hours of data. This indicates that task-specific model adaptation may be an effective approach in low-resource speech emotion recognition problems.