Distributed joint-distribution testing improves concept drift detection at scale
Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Machine learning models used in large e-commerce systems can become less accurate over time because the data they see changes, a problem called concept drift. The authors studied how well different mathematical tests detect changes in large, complex datasets. They found one particular test that works well even on huge data with many features, running efficiently on distributed systems. This helps companies spot when their models might need retraining to keep working well.
What this means in practice
- •For industrial data engineering teams: Detect concept drift accurately in very large-scale e-commerce data using scalable distributed joint-distribution tests on Apache Spark.
- •For cloud platform operators: Design sampling strategies that avoid statistical failures in drift detection due to ID-like features in massive datasets.
Authors
Cagdas Pullu, Mahmut Emir Arslan, Bugra Balkac, Aylin Ondersev Balta, Cihangir Celal Palaci, Fikri Cem Yilmaz, Altan Cakir
Abstract
Concept drift threatens production machine learning, yet the empirical behavior of multivariate two-sample drift detectors at scale remains under-characterized. Existing benchmarks rarely address the hundreds of millions of rows and high-cardinality features typical of industrial-operational datasets. We evaluate five multi-column two-sample tests (marginal, projection-based, and kernel embedding methods) across three complementary environments: the Harvard Dataverse, a validated Failing Loudly reproduction (mean absolute error between 0.030 and 0.053), and a novel synthetic-injection benchmark on the 137.5-million-row Trendyol collection-ranking feature table. Testing four drift types across two severity-scope regimes, we demonstrate that distributed Maximum Mean Discrepancy with Random Fourier Features on Apache Spark scales robustly. Averaged over the four drift types in the strong regime and under a calibrated threshold, it achieves a Pearson correlation of r = 0.940 with expected drift magnitude, an 80.4% true positive rate, and a 3.2% false positive rate. Conversely, the per-dimension Kolmogorov-Smirnov test failed due to statistic saturation from ID-like columns under asymmetric sampling, establishing a critical constraint for large-scale sampling design. At weak configurations (realized-flip fractions of at most 0.57%), detectors struggled to reliably discriminate, highlighting the need for future intensity-grid power analyses to distinguish fundamental sensitivity bounds from scalable threshold shifts.