Zero flux method compares complex discrete distributions accurately
Zero Flux: Flow-Based Comparison of High-Dimensional Discrete Distributions
Machine Learning
Summary
Comparing two complex sets of data where items have many characteristics is very hard because the number of possible combinations grows very fast. The paper extends a recent way to compare continuous data by looking at vector fields to discrete data using a new idea called the Zero Flux criterion. This criterion checks small local differences in probability flows between data points to determine if the two data sets are the same. The authors also show how to estimate this difference from samples reliably and demonstrate that their method works well on both made-up and real-world data.
What this means in practice
- •For data engineering teams: Detect and track changes in large-scale categorical datasets by measuring local shifts in distribution with the Zero Flux criterion.
- •For machine learning engineers: Identify subtle dependency shifts in high-dimensional categorical data to improve model robustness and update triggers.
Authors
Leyang Wang, Yakun Wang, Song Liu, Taiji Suzuki
Abstract
Comparing two high-dimensional discrete distributions has always been a challenging task due to the exponentially growing state space and complex changes in interactions. A recent work suggests comparing distributions through a vector field trained using flow matching between two continuous distributions. The resulting vector field at mid-point vanishes if and only if two distributions identical. However, such a flow-based criterion does not naturally apply to discrete distributions. We extend this principle to the discrete domain and introduce the \emph{Zero Flux} criterion, a discrepancy based on local probability fluxes. Under independent coupling, we show that all local probability fluxes vanish at the midpoint if and only if two distributions are the same. This discrepancy decomposes the joint distributional difference into smaller, local contributions and can be efficiently estimated from samples. We establish finite sample error bounds for our estimator. Experiments on synthetic and real categorical data demonstrate reliable recovery of sparse dependence signals and stable tracking of distribution shifts in high dimensions.