Fairness aware experiments improve reliable results across groups
Fairness-Aware Design for Contextual Experiments: Guaranteeing Reliability and Equity in Heterogeneous Subgroups
Artificial Intelligence
Summary
Designing experiments that work well for different groups of people at once is really hard. The authors created a method called F-CTSD that ensures experiments treat all groups fairly while still giving trustworthy results using fewer participants. Their approach balances fairness and efficiency and proves it works correctly with solid math. Tests show it beats older methods by reducing fairness mistakes while needing fewer samples.
What this means in practice
- •For healthcare data teams: Design clinical trials that ensure fair and reliable treatment effect estimates across patient subgroups.
- •For marketing analytics teams: Conduct promotional experiments that fairly evaluate outcomes across diverse customer segments with fewer test samples.
Authors
Guangyan Gan, Ling Zhang, Yanhua Cheng, Yongxiang Tang, Kaiyuan Li, Xialong Liu, Peng Jiang
Abstract
Experimental design is critical for evidence-based decision-making in healthcare, marketing, and public policy. However, designing efficient experiments across heterogeneous subgroups presents significant challenges. Existing methods often optimize for statistical power or overall sample efficiency, overlooking crucial fairness considerations across these different subgroups. To address this gap, we introduce a Fairness-Aware Contextual Track-and-Stop Design (F-CTSD) algorithm. The proposed F-CTSD algorithm provides statistical guarantees on subgroup fairness while minimizing required sample sizes. We quantify the fairness-efficiency trade-off and derive the exact sample complexity for the proposed F-CTSD algorithm under its fairness constraints. We further theoretically prove that the proposed F-CTSD algorithm consistently produces accurate treatment effect estimates even under fairness requirements, enhancing statistical reliability. Numerical experiments show that the proposed F-CTSD algorithm outperforms existing methods, achieving higher sample efficiency while reducing subgroup fairness violations by 4.95%.