Fairness-Aware Mixture-of-Experts via Subgroup Reweighting and Gate Regularization
2026-08-24 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how deep learning models can be unfair because they perform differently on groups like genders or ages due to imbalanced training data. They found that existing methods either learn fairness and predictions together but struggle with diverse groups, or learn them separately but may not align well. To fix this, the authors proposed a new method using a Mixture-of-Experts system that routes data to specialized parts while correcting biases by reweighting subgroup data and keeping the routing balanced. Their approach not only improves fairness but also helps understand how the model treats different subgroups without hurting overall performance.
Deep learningFairnessMixture-of-ExpertsSubgroup imbalanceReweightingGate entropy regularizationRoutingPerformance disparitySensitive attributesAdversarial training
Authors
Sunhee Hwang
Abstract
Deep learning models often produce performance disparities across demographic groups, due to the training data imbalance with respect to sensitive attributes such as gender or age. To address this problem, existing work has explored fair representation learning, data re-sampling, and adversarial training, which can be broadly categorized into two main approaches. Single-stage methods typically learn a shared representation for fairness, but often struggle to handle heterogeneous subgroup distributions. Two-stage methods learn representations separately from the final prediction task, which can lead to misalignment between fairness objectives and downstream predictions. We identify routing-induced bias, a failure mode in which subgroup imbalance drives the gating network to route subgroups onto a few experts, and propose an end-to-end Mixture-of-Experts (MoE) framework that corrects it. Specifically, we apply subgroup reweighting to correct data imbalance, and introduce gate entropy regularization to prevent routing from collapsing onto subgroup attributes, keeping expert utilization both balanced and interpretable. Beyond improving fairness, the routing distribution offers an interpretable view of how subgroups are allocated across experts. Experimental results demonstrate that the proposed approach improves fairness while maintaining competitive predictive performance.