Fusion system boosts emotion detection from partial multimodal data
Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition
Artificial Intelligence
Summary
Recognizing people's emotions from different signals like voice and facial expressions is hard when some data is missing or incomplete. The authors developed a new method that can combine whatever emotional clues are available, even if some signals are missing, to better identify a wide range of emotions using natural language labels. They do this by creating a library of typical emotional patterns across different signals to guide how data is fused. Their tests show this approach works better than previous methods when complete data isn't always available.
What this means in practice
- •For multimodal ai developers: Build emotion recognition tools that can still label emotions accurately when some sensor data is missing or incomplete.
- •For customer experience teams: Improve sentiment analysis in environments where full biometric data cannot be collected due to privacy or hardware limits.
Authors
Yichi Zhang, Shenyue Wang, Jing Luo, Chunyang Yu, Xinyu Yang
Abstract
Open-vocabulary multimodal emotion recognition (OV-MER) aims to generate open natural-language emotion labels from multimodal affective cues. In real-world scenarios, however, complete and synchronized modal data are difficult to obtain due to limitations of acquisition devices and user privacy constraints. Existing OV-MER methods are largely designed for full-modal inputs, and fail to perform effective feature fusion under modal missing conditions. Meanwhile, current fusion approaches designed for incomplete modalities mainly focus on fixed-label recognition context, and cannot satisfy the demand for fuse emotional cues guided with arbitrary emotion semantics in OV-MER context. To tackle these challenges, this paper proposes an Affect-Prototype-Conditioned Fusion (APCF) framework for incomplete open-vocabulary emotion recognition. As a candidate-free generative framework, APCF extends modal contribution learning to scenarios guided by arbitrary emotional semantics. Specifically, we construct an affect-prototype library to explicitly model multimodal contribution characteristics corresponding to diverse emotions, which provides dynamic constraints for modal fusion under different emotional semantic perspectives. Conditional retrieval and feature aggregation are conducted based on available modal features. The refined fused affective representations are then fed into an LLM decoder to produce open-vocabulary emotion labels. Experiments on the OV-MERD+ and MER-FG datasets demonstrate that APCF substantially outperforms state-of-the-art baselines.