Models and sensors compared for wearable emotion recognition accuracy
From Stress to Affect: Multimodal Deep Learning for Physiological Emotion Recognition Across Wearable Sensor Modalities
Machine Learning
Summary
The authors looked at how well different computer models can recognize emotions by using body signals collected from wearable devices. They tested three types of models and three ways to collect signals, either from the wrist, chest, or both, using two datasets. They found that no one model is best for all data, but using signals from multiple body parts helps improve accuracy. They also found that collecting data at a lower rate can still work well and save computing time.
What this means in practice
- •For mental health device makers: Use multimodal wearable data and selected deep learning models to improve emotion detection in stress monitoring devices.$Commercial implications: This paper supports building better emotion detection features in wearable health devices that can be marketed for stress and affect monitoring.
- •For human-computer interaction designers: Design interaction systems that rely on wrist and chest sensors combined to reliably detect user emotional states in real time.
Authors
Desta Haileselassie Hagos, Saurav Keshari Aryal, Legand L. Burge
Abstract
Physiological emotion recognition using wearable sensors has important applications in mental health monitoring, affective computing, and human-computer interaction. However, existing studies typically evaluate a single model, sensing configuration, or dataset, limiting our understanding of how these factors influence recognition performance. We present a comparative study of temporal deep learning architectures for physiological emotion recognition using two multimodal wearable datasets: WESAD and EmoWear. Bidirectional long short-term memory (LSTM), temporal convolutional network (TCN), and Transformer models are evaluated under wrist-only, chest-only, and multimodal sensing configurations using participant-independent leave-one-subject-out cross-validation (LOSO-CV). We also investigate soft-voting ensembles, sensor ablation, sampling frequency, and gradient-based saliency. The Transformer achieved the highest multimodal accuracy on WESAD (99.02% +/- 0.51%), whereas the LSTM achieved the best multimodal accuracy on EmoWear for both arousal (91.80% +/- 1.06%) and valence (89.96% +/- 0.36%). These results show that relative architecture performance depends on dataset characteristics rather than one architecture being uniformly superior. Multimodal sensing consistently outperformed wrist-only and chest-only configurations across both datasets. Sampling-frequency analysis showed that 4 Hz provides a practical operating point, with performance comparable to higher frequencies at substantially lower training cost. These findings provide guidance for selecting architectures, sensing modalities, and sampling frequencies for wearable physiological emotion recognition.