Performative prediction methods cope with missing outcome data
Performative Prediction with Selective Labels
Machine Learning
Summary
Machine learning models in areas like lending can influence how people behave, which changes the data these models see later. This paper studies what happens when labels (outcomes) are only available for certain groups, such as approved loan applicants. The authors show that training models only on these selective labels can mislead the process and prevent models from settling on stable, reliable predictions. They propose a way to adjust training that stays close to the best possible model even with limited label data. Their approach works well in tests involving fairness-aware lending decisions.
What this means in practice
- •For loan underwriting teams: Improve model retraining methods for credit decisions when loan repayment data is only available for approved borrowers.
- •For online marketing teams: Develop predictive models that account for behavior changes and partial feedback from only users who respond positively.
Tested on one dataset.
Authors
Giovani Valdrighi, Isabel Valera, Marcos Medeiros Raimundo
Abstract
Many social applications of machine learning exhibit performative effects: population behavior changes in response to deployed models. Performative prediction studies this interaction through a distribution map that relates each model to the population distribution it induces. One of the main results in this framework showed that repeated risk minimization (RRM), which updates models by retraining on the most recent data, can converge to a stable model that minimizes risk on its own induced distribution. However, existing analyses typically assume access to the complete distributions of features and labels after model deployment, ignoring the possibility of selective labels: observing labels only for the accepted subset of the population. In this work, we formalize performative prediction with selective labels and show that retraining only on observed data can misguide the retraining procedure and undermine the guarantees of convergence to a stable solution. We then propose a worst-case objective based on knowledge of a confidence interval on the probability of a positive label. Applying RRM to this objective permits us to remain within a bounded distance to the true stable point. Under a sensitivity assumption on the conditional label distribution, we further show how previously accepted data can tighten these confidence intervals over time. Experiments in a lending application with fairness regularization show that our robust optimization approach closely matches the performance of RRM with complete label access.