Learning Emotion from Motion: Kinetic Multi-Stream Skeleton Modeling with Metadata-Conditioned Weak Label Distributions
Computer Vision and Pattern Recognition
Summary
The authors address the challenge of recognizing emotions from body movements, which are often subtle and hard to label clearly. They propose a new method combining three different techniques: one using 6D rotations, another focusing on body parts and motion streams, and a third that accounts for uncertainty in emotion labels. By combining these approaches, their model improves emotion recognition accuracy and balance in testing. They also found that movement speed, bone positions, and limb motions are important for understanding emotions from body movement.
Authors
Sosuke Suzuki, Yijin Wei, Koichiro Kamide, Ran Dong, Haoran Xie, Chao Zhang
Abstract
Skeleton-based emotion recognition from body motion remains challenging because emotional expressions are often characterized by subtle dynamic and relational motion cues, and hard labels may not fully capture ambiguity among related emotion categories. For the DIEM-A task in the MMAC ACII 2026 Challenge, we propose a multi-branch skeleton-based emotion recognition framework that combines a 6D rotation-based branch, a part-aware kinetic multi-stream branch, and a metadata-conditioned weak label distribution learning (LDL) branch. The branches are trained independently and fused by a probability-level ensemble at inference time. In 10-fold leave-performer-out cross-validation, the proposed framework improves Accuracy from 0.271 to 0.366 and Macro-F1 from 0.252 to 0.353 over the rotation-based baseline. Explainability ablations show that velocity and bone streams, as well as arm and leg regions, provide important cues for recognizing emotional body motion.