Seer improves emotion understanding in cochlear implant speech
SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech
Sound
Summary
People using cochlear implants often have trouble recognizing emotions in speech because important vocal cues get weakened. The authors propose SEER, a new method that picks helpful example voices with the same emotion to make speech sound clearer and more emotional through cochlear implants. Unlike previous methods, SEER does not need matching recordings or emotion intensity labels to learn. Tests show SEER significantly improves emotion recognition for cochlear implant users.
What this means in practice
- •For hearing device engineers: Enhance cochlear implant speech processing to better preserve emotional cues using source-conditioned retrieval without needing parallel recordings.
- •For assistive technology developers: Incorporate emotion-aware speech enhancement into communication aids to improve emotional understanding for users with hearing impairments.
Authors
Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung, Chi-Chun Lee
Abstract
Cochlear implants (CIs) restore speech access but weaken cues needed for vocal emotion recognition. Prior CI-oriented enhancement requires parallel normal/strong recordings and intensity labels. We propose SEER, a retrieval-based framework that learns which same-emotion reference helps each source remain recognizable after CI processing. A source-conditioned retriever learns CI-aware utility from sampled emotional voice conversion outcomes, while uncertainty-guided exploration avoids exhaustive pair evaluation; neither parallel recordings nor intensity labels are required. SEER improves Source macro-F1 at N8 by 7.30 points on RAVDESS and 11.66 points on ESD, with significant ESD gains across N4/N8/N16. Sixteen-listener RAVDESS gains are significant across all conditions. Exhaustive analysis finds an aggregate benefit from stronger references but little effect from matching gender or content.