Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion
Computer Vision and Pattern Recognition
Summary
The authors created a system to detect feelings of ambivalence or hesitancy in videos by combining information from speech text, sound, and facial expressions. They first tested each type of data separately and found that text worked best alone. Then, they built a model that mixes all three types using special attention techniques, which improved accuracy by about 11%. Their approach shows that looking at how these different signals interact helps catch subtle cues better than using any single source. They also shared their code publicly for others to use.
Authors
Oussama Berhili, Yassine Ouzar, Larbi Boubchir
Abstract
We present a multimodal framework for Ambivalence/Hesitancy (A/H) recognition in video, developed for the ABAW11 challenge at ECCV 2026. The proposed approach fuses textual, acoustic, and visual modalities extracted from the BAH dataset using three pretrained encoders: F2LLM-v2-0.6B for transcripts (1024-d), WavLM-Large for audio (1024-d), and VideoMAE V2 for facial video (768-d). We first establish comprehensive unimodal baselines using classical classifiers (MLP, Random Forest, GBDT), each optimized via Optuna, and obtain a best unimodal Macro F1 of \textbf{0.6659} on the test set using text features alone -- substantially outperforming the zero-shot Video-LLaVA baseline (Macro F1: 0.2827). Building on these baselines, we propose a multimodal fusion architecture that combines bidirectional cross-attention across all three modalities with a Gated Multimodal Unit (GMU), with both architectural and optimization hyperparameters selected through a 50-trial Optuna search. This model achieves a Macro F1 of \textbf{0.7394} on the validation set, a relative improvement of 11.0\% over the best unimodal baseline, confirming that explicit cross-modal interaction captures complementary cues that no single modality provides in isolation. Final predictions on the official, unlabeled private test set are generated using this model and submitted according to the challenge protocol. Code is publicly available at https://github.com/yassineouzar/IUSD_AH/