AffectFuse: Cross-Task Feature Fusion with Temporal Modeling for Multi-Task Affective Behavior Analysis
Computer Vision and Pattern Recognition
Summary
The authors created a system to recognize emotions from face images by predicting different emotional cues at the same time. They used pre-trained models and adapted them after the main feature extraction step to focus on each specific task. For recognizing facial muscle movements (action units), they used a special approach that assigns experts to different units instead of handling them all at once. Their tests showed that this strategy works well without needing to train a new big model from scratch.
Authors
Dipit Saha, Mohammad Raihan Rashid, Shah Mohammad Abdul Mannan, Ahnaf Tahmid, Md. Mehedi Hasan
Abstract
Affective behavior recognition in the wild requires joint prediction of continuous valence-arousal, categorical facial expression, and multi-label action units from unconstrained face images. We present our system for the Multi-Task Learning (MTL) track of the 11th Affective Behavior Analysis in-the-wild (ABAW) competition on s-Aff-Wild2, the static selected-frame version of Aff-Wild2. The method focuses on post-encoder adaptation: frozen AffectNet-supervised backbones provide multi-resolution features, while task-specific temporal heads and cross-task fusion modules select the useful signals for each target. For action-unit recognition, we adapt MAE-Face with Low-Rank Adaptation (LoRA) and use DISFA through per-unit expert routing rather than direct sequential transfer. Ablations over backbone, temporal, fusion, and AU-adaptation choices define the final configuration. The final system obtains P = 1.7302 on the official validation split, showing that post-encoder adaptation and task-wise modeling choices provide a strong MTL pipeline without training a new large-scale face foundation model.