Benchmark gathers 19 public datasets for credit default prediction
PaMIR: Open Benchmark of Public Credit-Default Datasets
Machine Learning
Summary
Credit default prediction is hard when data labels are missing or delayed. The authors created PaMIR, a public benchmark collecting 19 datasets with over a million records from nine countries. This helps test models in realistic settings where labels come late and data arrives in order. They also provide tools to fairly measure model accuracy and test synthetic data generators without overlap. This benchmark aims to improve credit risk prediction research with more transparent and comprehensive data.
What this means in practice
- •For credit risk teams: Test credit-default prediction models on a large, diverse, and label-delayed public benchmark to evaluate real-world performance.
- •For synthetic data developers: Evaluate synthetic data generators against a challenging, non-overlapping benchmark to measure training data quality without leaking test information.
Authors
Mikhail Liashkov, Ilyas Varshavskiy, Shuhratjon Khalilbekov, Azizjon Azimi, Bonu Boboeva
Abstract
We release PaMIR (Public Arrival-ordered Measurement for Inference in Risk), an open benchmark for credit-default prediction when labels are scarce and arrive late. The field's reference benchmark studies use eight datasets each, only two or four of them public. PaMIR brings together 19 public datasets with binary default labels -- 1.24M loans, firms and card accounts from nine countries -- rebuilt from pinned source snapshots by one leakage-audited recipe and never redistributed; to our knowledge it is the one of its kind as of today. Every model is a single function, scored under a repeated i.i.d. split and a label-delayed stream in which each application is scored on arrival, with AUC reported by label budget; fleet means are withheld unless every dataset is scored. A synthetic-data harness tests generated training rows without letting a generator see held-out rows. This report describes release 0.4.0 of this living benchmark.