Improving machine learning with localized minority data imbalance correction
Learning under Localized Minority Imbalance
Machine Learning
Summary
Some machine learning systems struggle when data about rare but important groups is missing unevenly in certain areas. The authors show that common fixes assume missing data happens evenly, which isn’t true in many cases, and this can cause errors. They propose splitting the data into smaller parts and training separate models to handle these tricky areas better. Their method works better in tests than popular alternatives.
What this means in practice
- •For business data teams: Improve predictive models for risk assessment where certain rare business outcomes are unevenly missing from records.
- •For fraud detection teams: Enhance fraud classifiers by correcting localized underrepresentation of certain fraud types in training datasets.
Authors
Amin Hosseininasab, Steven M. Shugan
Abstract
Class-imbalance methods implicitly assume that the minority class is uniformly undersampled relative to the majority class. However, in many real-world settings, minority instances may be disproportionately under-observed in certain regions of the feature space. For example, small businesses that go bankrupt may disappear from records, while those that survive remain visible, making bankruptcy appear less common among small firms than it actually is. This gives rise to localized minority imbalance (LMI), a challenge that is often overlooked and extends beyond general class-count imbalance. We show that under LMI, existing imbalance mitigation techniques can fit observation-induced biases in the training data and generalize poorly to under-observed regions of the true minority distribution. To address this, we propose a tree-based stratified approach that recursively partitions the feature space with the goal of reducing within-stratum LMI distortion. For each resulting stratum, we pair its majority instances with the full observed minority set and train a base classifier to create an ensemble. Extensive experiments over benchmark tabular datasets simulated with LMI show that our stratified ensembling approach outperforms popular and state-of-the-art imbalance mitigation techniques. We also introduce a gold-standard evaluation protocol that uses unbiased test sets, and demonstrate that conventional hold-out evaluation from the same LMI-biased data can substantially mislead performance. Overall, our results highlight that the cause of imbalance is as important as the correction method.