Not all Jensen-Shannon Divergence Estimators are Equal
2026-06-15 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explain that Jensen-Shannon divergence, a common way to measure how close synthetic tabular data is to real data, can give very different results depending on how it's calculated. They found that simple methods ignoring relationships between data features can underestimate differences, while more complex methods are sensitive to how the estimation is done, especially when data is imbalanced or has many features. To fix some of these issues, the authors provide a mathematical correction and argue that the exact estimation method should always be clearly stated. They also offer practical advice and a tool to help others measure Jensen-Shannon divergence more reliably.
Jensen-Shannon divergenceSynthetic dataTabular dataEstimator biasClass imbalanceDimensionalityMarginal estimatorsClassifier estimatorsPrior shiftEvaluation protocol
Authors
Alba Garrido, Alejandro Almodóvar, Mar Elizo, Patricia A. Apellániz, Santiago Zazo, Juan Parras
Abstract
The Jensen-Shannon divergence is widely reported as a scalar measure of fidelity for synthetic tabular data. Yet, in practice, it is estimated from finite samples using protocols that are often underspecified. This creates a measurement problem. Although the population divergence is well defined, the empirical value depends on the estimator family, sampling protocol, calibration, dimensionality, and class balance. We show that different protocols can yield non-comparable values: marginal-based estimators ignore dependencies in the joint distribution and can severely underestimate divergence, while classifier-based estimators capture joint structure but exhibit strong estimator dependence. We systematically study this behavior across controlled settings with reference divergences and real-world synthetic tabular benchmarks. Our analysis reveals dependence blindness in marginal estimators, prior-shift bias under class imbalance, and estimator sensitivity in high dimensions. To address prior shift, we derive a closed-form posterior correction for classifier-based Jensen-Shannon estimation. Our results show that empirical Jensen-Shannon divergence values are inherently protocol-dependent, making explicit specification of the estimation procedure necessary for meaningful comparison. We provide practical guidelines and an open-source tool for estimator-aware Jensen-Shannon evaluation.