Multimodal system improves aspect sentiment analysis on social media
Multimodal Aspect-Level Sentiment Analysis Based on Gated Noise Filtering and Emotion-Relevance Interaction
Multimedia
Summary
Understanding people's feelings about specific parts of a product or topic in social media posts is tricky, especially when posts include both text and pictures. The authors address two big problems: unwanted irrelevant details in text or images that confuse analysis, and mismatches between the feelings expressed in words versus pictures. They created a model that filters out irrelevant information and better matches emotions from text and images to improve detecting sentiment about each aspect. Tests on popular Twitter datasets showed their method performs better than previous approaches.
What this means in practice
- •For social media analytics teams: Improve detection of user opinions about specific aspects in posts combining text and images, enhancing market or brand insights.
- •For customer experience managers: Analyze multimodal feedback to get more precise sentiment insights on product features by reducing irrelevant noise in text and images.
Authors
Chen Huang, Liangwei Guo, Yamin Li, Yan Zhang, Chao Yang, Li Yang, Jianhua Song
Abstract
Multimodal Aspect-Based Sentiment Analysis (MABSA) infers fine-grained sentiment polarity toward specific aspects by jointly modeling text and images. Despite progress in cross-modal fusion, two challenges remain in multi-aspect settings: (1) multimodal noise, where aspect-irrelevant content distracts sentiment learning; and (2) weak cross-modal sentiment alignment, as visual evidence can be ambiguous and textual--visual sentiments may conflict, limiting multimodal complementarity. To address these issues, we propose a Gated Noise-filtered Sentiment-Relevance Interaction (GNSRI) framework. It employs a gated noise-filtering module to suppress sentiment-irrelevant features and enhance aspect-aware sentiment cues, and a sentiment-relevance interaction module to capture consistent and conflicting cross-modal signals at micro and macro levels. Finally, a learnable decision fusion mechanism adaptively combines predictions from textual, visual, and cross-modal branches at the aspect level. Experiments on public MABSA benchmarks show that GNSRI outperforms state-of-the-art methods, improving accuracy by 1.94\% and 2.06\% on Twitter-2015 and Twitter-2017, respectively.