Policy learning method controls treatment harm risk without assumptions
Conformal Policy Learning with Distribution-Free Safety Guarantees
Machine Learning
Summary
Sometimes deciding who should get a treatment can accidentally hurt people. This paper presents a method that helps decide when to give treatment so that the chance of harm is kept low, no matter how different or weird the data is. The method uses special statistical tests to control risk without assuming anything about how outcomes behave. It works for both randomized tests and observational data and aims to make the best possible decisions while keeping people safe.
What this means in practice
- •For public health analysts: Make treatment assignment decisions in policy studies that limit the chance of harming individuals using finite-sample safety guarantees.
- •For clinical trial teams: Assess new medical treatments by controlling the probability of causing harm during patient assignment without relying on outcome model assumptions.
Authors
Ying Jin, Naoki Egami
Abstract
Policy learning aims to determine who should be treated based on individual characteristics. In high-stakes settings such as medicine and public policy where safety is a central concern, improving the average outcomes alone may not be sufficient: decision makers may also seek to protect individuals from harm, in line with the Hippocratic principle of ``do no harm.'' In this paper, we propose \textit{conformal policy learning} (CPL), a policy learning procedure with a new distribution-free safety guarantee that controls the probability of assigning treatment to an individual who would be harmed relative to control. CPL views each treatment decision as testing a hypothesis of counterfactual harm and assigns treatment by thresholding conformal p-values. These p-values use observable proxies and selective calibration to address the challenge that the potential outcomes under comparison are never simultaneously observed. For randomized experiments, under standard exchangeability conditions, CPL provides finite-sample safety guarantee at a user-specified level, without imposing any outcome modeling assumptions. Moreover, when the outcome model is consistently estimated, CPL achieves asymptotically optimal welfare subject to the safety constraint. In observational studies, CPL with learn-then-balance weights achieves doubly robust safety guarantees. We evaluate CPL through extensive simulations and apply it to an empirical study of AI-powered interventions designed to reduce conspiracy beliefs.