Model Multiplicity and Predictive Arbitrariness in Recidivism Risk Assessment
2026-06-01 • Machine Learning
Machine LearningComputers and Society
AI summaryⓘ
The authors studied a system used to predict whether released inmates will reoffend, addressing concerns about different models giving different predictions for the same person. They created a large dataset by converting legal rules into labels, then built simpler models that were accurate, fairer across groups, and sensitive to rehabilitation progress. They analyzed how different accurate models can disagree on predictions and found that, in practice, these disagreements are less severe than theory suggests. They showed that picking the lowest risk prediction across models helps reduce arbitrary decisions.
recidivism risk assessmentpredictive multiplicitymachine learning modelserror-rate disparitiesinterpretable modelsrisk scoresmodel agreementpost-release outcomesalgorithmic fairnessstructural diversity
Authors
Ashwin Singh, Carlos Castillo
Abstract
Prediction tasks over individual futures, which are inherently noisy, often admit multiple similarly accurate models. When these models produce different predictions for the same individual, they raise concerns of arbitrariness in decision-making. How severe can this arbitrariness be, in theory and in practice? How can it be resolved to support high-stakes risk assessment? We address these questions through a study of a machine learning-based decision support system for recidivism risk assessment that has been in use for over 15 years. By translating complex legal rules into an algorithm for labeling post release outcomes (recidivist or non-recidivist), we first construct a dataset of thousands of inmate releases. Using this dataset, we learn interpretable models that improve predictive performance, reduce error-rate disparities between groups, and ensure that rehabilitative progress lowers risk scores. Next, we study predictive multiplicity, by first deriving a tight lower bound on the expected predictive agreement of any finite set of models over a dataset, and then by evaluating the extent to which structural diversity (e.g., different model coefficients) within this set translates to predictive multiplicity (i.e., different predictions for the same individual). Our experiments indicate that the existence of many similarly accurate models with comparable error-rate disparities does not necessarily translate into severe predictive multiplicity. Empirically, similarly performant models can exhibit substantially higher predictive agreement than worst-case theoretical guarantees suggest. We find that a simple policy that assigns each inmate the lowest risk among these models is effective for addressing predictive arbitrariness.