When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification
2026-08-17 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors show that testing methods for handling class imbalance on just one dataset can give misleading results. On the popular credit-card fraud dataset, tuning classification thresholds did not improve a Random Forest model’s performance. However, when they tried 45 different tasks with various imbalance levels, tuning helped a lot, especially for Random Forest models. They also found that the benefit of tuning changes depending on how imbalanced the data is, and common checks for calibration errors don’t reliably predict if tuning will help. The authors provide all their code and data to allow others to reproduce and extend their work.
class imbalanceRandom Forestthreshold tuningF1 scorecross-validationSMOTEcalibration errorimbalanced classificationnested validationmodel evaluation
Authors
Diyorbek Musaev
Abstract
Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.