Online prediction method improves fairness in adaptive decision making
Counterfactual Online Conformal Prediction Under Adaptive Logging
Machine Learning
Summary
When computer systems make predictions that influence decisions, those decisions affect the data used to check if predictions were right. This feedback can cause standard ways to estimate prediction accuracy to be biased, especially for actions chosen less often. The authors point out this problem and offer a new method that adjusts for the bias by weighting data according to how likely each action was chosen. They also propose a more accurate version that reduces errors from both outcome prediction and action choice models. Their method comes close to a theoretical limit on how well these predictions can perform and works better in tests without losing precision.
What this means in practice
- •For online marketing teams: Improve confidence estimates in personalized ad placement where past ads affect future user responses and data collection.
- •For financial portfolio managers: Better quantify uncertainty in algorithmic asset rebalancing decisions that depend on previous trades influencing market responses.
Authors
Xinyu Qiao, Yichen Lin, Kaihong Ji, Xue Wang, Tao Yao
Abstract
Online conformal prediction can fail when predictions shape actions and actions determine which outcomes enter calibration. Standard adaptive methods may retain marginal coverage while systematically miscovering the counterfactual outcomes of rarely selected actions. This paper formalizes the failure through counterfactual coverage and introduces Propensity-Weighted Online Conformal Prediction, an inverse-propensity-weighted recursion that debiases calibration. A doubly robust variant further reduces nuisance bias to the product of outcome-model and propensity errors. Under positivity, the resulting coverage rate matches an information-theoretic lower bound up to logarithmic factors. Experiments on synthetic decision tasks, open bandit data, and financial rebalancing show that PW-OCP and DR-OCP improve counterfactual coverage and downstream regret without sacrificing prediction-set sharpness.