Vista helps machines understand mixed emotional signals in context
VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict
Artificial Intelligence
Summary
Sometimes different parts of a person’s expression can show conflicting emotions, like a quiet voice but a smile. The authors developed VISTA, a method that helps computers figure out what these mixed signals mean by considering what’s important to the person and the situation. This makes emotion recognition better, especially when signals don’t agree. VISTA uses a shared model that improves understanding by combining clues with an interpretation of value and expectations about feelings.
What this means in practice
- •For human-computer interaction developers: Improve emotional understanding in voice assistants and social robots by interpreting mixed emotional signals based on user context and values.
- •For mental health technology teams: Enhance emotion monitoring tools by better resolving conflicting emotional cues to provide more accurate assessments of users’ feelings.
Authors
Jiale Dai, Liuxian Ma, Xiaoke Niu, Wenjing Zhang, Huiying Zhao, Zhaoxiang Liu, Shiguo Lian, Guojie Song
Abstract
Conflicting emotional cues can be individually valid: a subdued voice may reflect a blocked goal while a smile satisfies a social obligation. Their interpretation depends on what the event means to the person. We introduce VISTA (Value-Informed Semantic Trust Arbitration), a learned seven-field appraisal interface that conditions modality arbitration on concerns, event relations, and expression conditions while retaining a joint-evidence residual. A log-odds decomposition separates emotion expectation from cue diagnosticity, motivating an interface that lets appraisal change how evidence is interpreted. With a shared Qwen2.5-Omni-7B backbone and matched training examples and steps, VISTA reaches 64.5% conflict accuracy on CA-MER, improving on modality gating by 2.5 percentage points on conflict and 0.2 on consistency. Shuffling appraisal across scenes or removing its decision connection reduces this benefit. A common frozen-backbone probe reaches 0.600 macro CCC for appraisal readout, compared with 0.505 for emotion-only fine-tuning. Evaluations across five benchmarks connect recognition under increasing conflict with appraisal readout and downstream decision use. Together, the analyses and experiments support scene-specific appraisal as an intermediate representation that helps interpret conflicting emotional evidence.