CIFA: Contextual-Intersectional Fairness Auditing for Hidden Subgroup Discovery in Face Analysis
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors point out that common ways to check fairness in vision models focus on accuracy and broad demographic groups but miss how other factors like lighting, blur, and accessories affect performance. They introduce a new method called CIFA that looks at how these context factors combine with demographics to find hidden groups where the model does poorly. Testing on gender classification with popular models and datasets, they show that ignoring these combined factors can hide big fairness problems. They also tried fixes but found none fully solved the issues across all tests. Their work highlights the need to check for these combined fairness problems systematically.
fairness evaluationcomputer visiondemographic subgroup analysiscontextual factorsintersectionalityResNet-50Vision Transformer (ViT)FairFace datasetCelebA datasetUTKFace dataset
Authors
Nazia Aslam, Khalid Adnan Alsayed, Thomas B. Moeslund, Kamal Nasrollahi
Abstract
Fairness evaluation in computer vision commonly relies on aggregate accuracy and demographic subgroup analysis. However, visual models are also sensitive to contextual factors such as illumination, blur, image quality, facial accessories, and appearance attributes. These factors may interact with demographic characteristics, producing hidden subgroups in which performance degrades substantially despite strong aggregate accuracy and apparently acceptable demographic fairness. To address this, we propose the Contextual-Intersectional Fairness Auditing Framework (CIFA), a structured framework for identifying subgroup vulnerabilities arising from interactions between demographic and contextual attributes. CIFA performs demographic, contextual, and contextual-intersectional auditing, followed by worst-group discovery to identify and rank the most vulnerable attribute combinations. We evaluate CIFA on gender classification using ResNet-50 \cite{he2016deep} and ViT-B/16 \cite{dosovitskiy2020image} across FairFace \cite{Karkkainen2021}, CelebA \cite{Liu2015}, and UTKFace \cite{Zhang2017}. Our results show that aggregate accuracy and demographic-only evaluation can mask substantial contextual-intersectional disparities. We further assess several established mitigation strategies through an audit--mitigate--reaudit protocol and find that, although some worst-group disparities are reduced, no single strategy consistently eliminates them across datasets and architectures. These findings establish contextual-intersectional auditing as an important component of fairness evaluation and provide a reproducible framework for discovering, prioritizing, and reassessing hidden subgroup risks in face analysis systems.