Graph method improves emotion recognition in artworks over vision language models
Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageMachine Learning
Summary
Understanding emotions in artworks is hard because feelings can be very personal and subtle. The authors created a new method that breaks down artworks into parts like colors, brushstrokes, and shapes, then connects these parts in a graph to predict the emotions the art expresses. This approach did better than using large vision-language models alone, especially when emotions were organized by general feelings like positivity or energy. Their work shows that explicitly looking at how different art features relate helps computers better guess the emotions in artwork.
What this means in practice
- •For digital art platforms: Enhance automatic tagging of artwork emotions by analyzing visual and semantic features with graph models for more accurate emotional labels.$Commercial implications: Provides improved emotion tagging for art marketplaces and galleries to offer better search and recommendation features.
- •For multimedia content creators: Use explicit emotional understanding of artwork elements to guide artistic design choices or personalized mood-based content creation.
Authors
Ngoc-Duy Mai, Ha-Thanh Nguyen, Hideaki Takeda
Abstract
Emotion recognition in artworks remains challenging because affective meaning is often subjective, implicit, and shaped by relationships between visual form and semantic associations. While vision-language models provide strong multimodal representations, direct image-to-emotion prediction may overlook explicit relations among formal, semantic, and affective cues that support emotional interpretation. This paper proposes Visual-Semantic Affect Graphs (VSAG), a graph-based framework that represents artworks through interpretable feature nodes, including color, brushstroke, composition, light-shadow, line quality, and extracted visual entities. The graph learns feature-emotion associations for dominant emotion prediction, with valence-arousal structure modeled through an optional hierarchical auxiliary affect module. We evaluate VSAG and VSAG-TopK against direct VLM inference using Qwen3-VL and InternVL3.5 on EmoArt-5k. Results show that direct VLMs provide useful but modest baselines, while graph-based models improve dominant emotion prediction. In particular, the hierarchical valence-arousal variant of VSAG achieves the best overall performance, improving accuracy, macro-F1, and weighted-F1 over both direct VLM inference and the emotion-only VSAG baseline. Per-class results indicate that graph-based models and VLMs capture partly complementary signals, while valence-arousal structure is most useful when integrated as a coarse-to-fine affect hierarchy rather than as a simple post-hoc correction. These findings suggest that explicit visual-semantic structure can support both prediction and interpretation in artwork emotion recognition.