Evidential transformer improves emotion recognition with uncertainty handling
Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition
Machine Learning
Summary
Recognizing emotions from multiple sensors like facial expressions and voice can be tricky when the signals conflict or get noisy. The authors present a new method that measures how uncertain each sensor’s input is and combines them carefully to avoid wrong confident guesses. Their approach improves accuracy and reliability in detecting emotions across several test datasets. It also flags which predictions might be wrong, which can help in sensitive applications.
What this means in practice
- •For speech recognition engineers: Improve voice-based emotion recognition systems by integrating uncertainty-aware fusion to reduce misclassification in noisy conditions.
- •For clinical monitoring teams: Flag uncertain emotion predictions in patient monitoring to enable focused review and reduce risk in safety-critical decisions.
Authors
Muhammad Usama, Yeoreum Jung
Abstract
Softmax-based multimodal emotion recognition methods often assign high confidence even when sensor data are corrupted or contradictory, for example when facial expressions suggest positive valence while vocal prosody indicates negative arousal. We present an Evidential Multimodal Transformer with Dempster-Shafer Fusion (EMT-DS) that addresses this problem through formal uncertainty quantification. EMT-DS models predictions as Subjective Opinions (Dirichlet distributions), separating epistemic uncertainty (lack of evidence) from aleatoric uncertainty (sensor noise). Modality-specific transformer encoders produce belief mass distributions that are fused with Dempster's Rule of Combination, which normalizes the joint belief mass by a conflict coefficient 1/(1-c), so that contradictory evidence is explicitly quantified as conflict and the fused prediction becomes diffuse rather than being forced into an unjustified consensus. Under subject-independent and speaker-independent evaluation on four benchmarks, EMT-DS achieves 84.3% accuracy on IEMOCAP versus 81.7% for the best baseline (p < 0.01), 82.4% on DEAP under leave-one-subject-out evaluation versus 78.1%, and 96.8% on the highly imbalanced KEMDy20, and it improves accuracy on CMU-MOSEI by 2.4 points over the strongest evidential baseline. In controlled ablation, it reduces Expected Calibration Error from 0.153 (softmax transformer) to 0.042. It maintains 71.2% accuracy under severe Gaussian noise on audio features (sigma = 1.0), where attention baselines drop to 45.3%, and degrades gracefully when entire modalities are absent. The fusion step has O(MK^2) complexity, compared with O(M^2 d^2) for cross-modal attention. The learned conflict metric also flags likely errors: reviewing 20% of samples captures 80% of misclassifications, which supports selective prediction in clinical and safety-critical systems.