Spatiotemporal model improves facial expression recognition in videos
GM-GReFEL: Geometry-Guided Spatiotemporal State-Space Learning for Robust Dynamic Facial Expression Recognition
Computer Vision and Pattern RecognitionMachine Learning
Summary
Recognizing facial expressions in videos is challenging because analyzing long clips requires a lot of computing power and the labels used for training often have errors. The authors designed a new method that looks at different parts of a face separately over space and time to reduce computing cost. Their system also trusts facial geometry more when it is unsure about a prediction, improving accuracy. Tests show the approach performs competitively on multiple facial expression datasets while keeping speed close to standard methods.
What this means in practice
- •For mobile app developers: Create apps that can analyze facial expressions in video calls efficiently by using a model that processes facial regions over time with reduced computing cost.
- •For security system engineers: Improve video surveillance systems with better detection of emotional states by combining facial geometry with uncertainty-aware classification.
Authors
Yudhistira Arditya Pratama, Yi-Zeng Hsieh
Abstract
Recognizing facial expressions in unconstrained video suffers from two coupled problems: self-attention grows quadratically in cost with clip length, and crowdsourced labels are noisy. We present GM-GReFEL, a spatiotemporal state-space network with a geometry-aware decision stage. Its Modulated GroupMamba backbone replaces a spatiotemporal Vision Transformer. Six channel groups are scanned separately, four over the frame plane and two over time, so the cost grows linearly with sequence length ($\mathcal{O}(N)$); after each layer, a Channel Affinity Modulation gate mixes the groups again. On top of the backbone, a GReFEL reliability module turns the normalized entropy of each prediction into a mixing weight between the classifier output and a vote from trainable geometric anchors, so that facial geometry counts for more when the network is unsure. The model is initialized by static-to-dynamic weight transfer and trained with a Triple Loss. In five-fold cross-validation, the best fold scores 66.57\%/76.46\% UAR/WAR on DFEW, 0.3 points or less below the strongest compared method, and 45.66\%/62.27\% on MAFW, ahead of all compared methods; on FERV39k the mean over five splits, 42.30\%/52.66\%, is competitive. The five-fold means are 64.07\%/74.54\% on DFEW, on par with recent large pre-trained models, and 38.90\%/54.37\% on MAFW, below the best of them. A paired $t$-test on DFEW indicates a modest benefit of temporal modeling (+1.61 UAR points, one-sided $p=0.031$). Attention stays on plausible facial regions, and latency is close to a ViT-B/16 baseline (47.4 vs.\ 46.8 ms per clip). Code: https://github.com/ardiytama/GroupMamba-DFER.