Rank based fairness reveals hidden bias in mortgage lending models
A Rank Graduation metric for Algorithmic fairness
Machine Learning
Summary
Credit scoring algorithms are often checked for fairness by comparing group averages, but this might miss unfair treatment of individuals. The authors propose a new method that looks at how prediction mistakes are ranked across different groups, linking fairness with accuracy and explanations. They tested this method on several common models using mortgage data and found that all models showed unfairness based on this new measure, especially tree-based models showing better fairness-accuracy balance. Their work highlights that unfairness is often due to underlying data differences, not just the choice of algorithm.
What this means in practice
- •For financial data teams: Evaluate and compare credit risk models with a rank-based fairness metric that reveals unfairness beyond group averages.
- •For machine learning engineers: Improve model selection by balancing predictive accuracy and a rank-based fairness measure that offers explainability via feature removal.
Authors
Dalia Atif, Paolo Giudici
Abstract
Fairness assessment in algorithmic decisions that affect individuals, such as credit scoring, often relies on parity measures calculated at the aggregate group level. Such measures may not reveal which individuals experience unfairness or which explanatory factors contribute to it. In this paper, we propose a rank-based framework that evaluates fairness through the distribution of model prediction errors, thereby linking fairness assessment with predictive accuracy and explainability. The framework combines Rank Graduation Fairness (RGF), its integrated measure AURGF, a centered Cramer--von Mises permutation test, and a feature removal procedure for fairness explainability. We evaluate the methodology using logistic regression, random forest, gradient boosting, and a multilayer perceptron. The simulation study shows that protected-group imbalance can reverse descriptive fairness comparisons, whereas the proposed inferential procedure correctly distinguishes fair from unfair mechanisms. Its application to HMDA mortgage data produces model rankings that differ from those obtained with classical fairness criteria. Tree-based models, rather than logistic regression, provide the strongest combination of predictive accuracy and rank-based fairness, while the fairness null hypothesis is rejected for all four models. The persistence of disparity across statistical, bagging, boosting, and neural network specifications, together with the feature removal results, indicates that the observed unfairness is not specific to a single algorithm or predictor, but is associated with group differences embedded in the characteristics of the lending data. These findings support a broader approach to trustworthy artificial intelligence that combines predictive accuracy, fairness measurement, statistical inference, and explainability.