Multimodal system improves emotional support conversations with fine-grained alignment
FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation
Computation and Language
Summary
Emotional support conversations are tricky because they involve understanding not just words but also feelings from different sources like images or audio. The authors propose a new system, FACES, that better matches and aligns these different kinds of information to give more thoughtful and appropriate support. Their system uses careful techniques to combine rewards from multiple feedback types and reasons about responses at a detailed level. In tests, FACES improved automated scores and was preferred by human judges compared to previous methods. It also worked well on different datasets, showing it can adapt beyond its original training.
What this means in practice
- •For chatbot developers: Enhance emotional support chatbots by integrating multi-source feedback for more relevant and empathetic responses.
- •For customer service teams: Improve automated support tools to better understand and respond to customers' emotional cues using multimodal alignment techniques.
Authors
Hang Ren, Peidong Wang, Yichen Gao, Wen Zhang, Jiayang Yu, Xiaocui Yang, Shi Feng
Abstract
In recent years, with advances in multimodal interaction technologies and the growing demand for intelligent emotional support, Multimodal Emotional Support Conversation (MESC) has emerged as an important task. However, existing systems still suffer from shallow responses and inappropriate support strategies due to weak semantic feedback, cross-dimension reward compensation, and coarse reasoning supervision. To address these challenges, we propose FACES, a fine-grained gated semantic alignment framework for MESC. Specifically, FACES integrates multi-source rewards to construct multidimensional semantic and functional alignment signals, employs a non-compensatory gated fusion mechanism to reduce cross-dimension reward offset, and leverages structured Chain-of-Thought (CoT) anchors to perform token-level credit assignment. Under a controlled response-length protocol, FACES improves the average MLLM-based evaluation score from 3.61 to 4.04 and achieves an 80.22% pairwise win rate over SOTA. Human evaluation under the same protocol yields a 75.3% preference rate. Results on three external datasets further support transferability. The code and dataset are available at https://github.com/OCT-ARR/arr.