Summary
Automated systems are used to make important life decisions, but choosing what to prioritize—privacy, fairness, or accuracy—can create tradeoffs. The authors conducted a study where people evaluated different decision scenarios involving these tradeoffs. They found that most people preferred human judgment because humans understand context and can consider things that are hard to measure. The idea of fairness was seen as more than just equal treatment—it included concerns about privacy and fairness in outcomes. This shows that designing automated systems needs to consider how people actually see fairness and harm in real situations.
What this means in practice
- •For policy makers: Design regulations and guidelines that reflect public values and preferences about fairness, privacy, and human involvement in automated decisions.
- •For product managers: Create automated decision-making tools that offer configurable tradeoffs catering to specific contexts and user expectations about fairness and privacy.
Authors
Rabeya Bosri, Anna Harbluk Lorimer, Afrida Hossain, Vasisht Duddu, Bailey Kacsmar
Abstract
Automated decision-making (ADM) systems are increasingly deployed in domains such as mortgage lending, prison sentencing, health insurance coverage, and hiring. Designing a responsible ADM system in such high-stakes domains requires ensuring privacy protection, fairness across demographic groups, and robustness against adversarial manipulation. However, prioritizing one of these objectives comes at the cost of another, forcing a choice as to which tradeoff to accept in a deployment. These tradeoffs explicitly or implicitly impact the life, safety, and fundamental rights of the people in a society, and thus, the perceptions and priorities of this population are needed before we can produce appropriate solutions. To this end, we conducted a quasi-experimental study (N = 777) in which participants evaluated four decision-making scenarios with controlled tradeoffs. Participants significantly preferred human decision-making (HDM) over ADM in three of four scenarios, emphasizing the value of human judgment, contextual understanding, and the ability to incorporate non-quantifiable factors. Furthermore, in terms of tradeoffs, our findings not only show that participants' preferences are highly context-dependent, but also that their perception of a specific objective, fairness, extends beyond formal definitions. Participants interpret fairness through multiple lenses, including privacy risks and susceptibility to manipulation, and view unfair or manipulated outcomes as failures of accuracy. Overall, our findings highlight the importance of context-aware and human-centered approaches when designing and governing ADM systems in high-stakes situations. Rather than purely technical objectives, it is essential to evaluate ADM systems based on how their tradeoffs align with specific expectations within a given domain, as well as with societal values and perceptions of harm and fairness.