Model predicts human disagreement to improve facial expression recognition
Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition
Computer Vision and Pattern Recognition
Summary
Facial expression recognition computers usually guess one emotion per face video clip, ignoring when people disagree on what emotion it shows. The authors propose a new approach that trains the model to predict not just one label but how often different people chose each emotion, capturing disagreements. Their model also estimates uncertainty and decides when it’s unsure enough to refuse a prediction. This method keeps accuracy while being better calibrated and more aware of ambiguous cases.
What this means in practice
- •For facial recognition developers: Improve expression recognition systems by representing annotator disagreement and enabling models to reject uncertain predictions for more reliable outputs.
- •For human computer interaction teams: Enhance user experience by deploying emotion recognition that quantifies ambiguity and uncertainty for adaptive interaction based on model confidence.
Authors
Yiming Wang, Frederick W. B. Li, Jingyun Wang
Abstract
Dynamic facial expression recognition (DFER) benchmarks such as DFEW provide multiple annotator votes per clip, yet most models collapse them to a majority label and cannot represent human disagreement at inference time. We propose a disagreement-aware DFER framework that trains directly on the raw annotator count vector using a Dirichlet-Multinomial likelihood. Unlike mean-only soft-label objectives, the proposed likelihood provides scale-sensitive supervision for the Dirichlet concentration while preserving the predictive mean. A separate ambiguity head predicts annotation entropy for unseen clips, and a monotone Chow-style reject rule combines predicted ambiguity, vacuity, temporal instability, and input quality for selective prediction. On DFEW, the method preserves recognition accuracy while reducing ECE by 30% and AURC by 15%, and predicted ambiguity reaches a Spearman correlation of 0.52 with the annotation entropy of test clips. The calibration and selective-prediction gains transfer to FERV39k and remain under identity- and movie-disjoint DFEW splits.