AI summaryⓘ
The authors developed a new system called MUPA²E that can understand emotions by looking at both facial videos and brainwave signals (EEG) together, using one combined model instead of separate ones. They tested this system on a dataset and found it worked best when combining both types of data. However, they noticed that some emotions were recorded for different lengths of time, which made the model rely on recording length to guess emotions. When they fixed this by making all recordings the same length, the system's accuracy dropped but still showed the method can handle very different kinds of signals in one model. Their work also shows it's important to control for factors like recording duration in emotion studies.
emotion assessmentfacial videoEEGmultimodal fusionattention backboneDMER datasetsignal processingclassification accuracyaffective computingrecording duration control
Authors
Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez
Abstract
Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~$30$ achieves the highest validation performance and a test accuracy of $70.07\%$. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.