Conformal test martingales have blind spots for some data shifts
Information-Geometric Blind Spots of Conformal Test Martingales
Machine Learning
Summary
This paper shows that a certain statistical method called conformal test martingales, used to detect changes in data streams, can fail to notice some types of shifts in data. The authors explain that if the method's scoring function produces the same output distributions for two different data sources, then the test cannot tell them apart. They mathematically characterize these invisible shifts and demonstrate with image data that a data distribution can change drastically without being detected by these tests. This helps us understand the limitations of current methods in detecting some kinds of changes in data.
What this means in practice
- •For statistical machine learning engineers: Design more robust sequential tests by avoiding conformity functions with large blind spots in detecting data distribution shifts.
- •For data quality assurance teams: Improve monitoring of data streams by understanding limitations of current tests that may miss subtle but critical distribution changes.
Tested on one dataset.
Authors
Johan Hallberg Szabadváry
Abstract
Conformal test martingales (CTMs) are e-processes for sequentially testing exchangeability via p-values produced by a conformity function $A$. However, any distribution shift that preserves the push-forward $P\circ A^{-1}$ leaves the p-value distribution invariant, rendering it invisible to every CTM based on $A$. We formalise this blind spot through the \emph{$A$-equivalence class} $[P]_A$ and show that, for any target distribution $T$, there is a unique closest element $Q^\ast\in[P]_A$ in KL divergence---the \emph{$A$-cryptic I-projection}---with explicit density $q^\ast(z) = t(z)\,p_S(A(z))/t_S(A(z))$ and a Pythagorean decomposition into detectable and cryptic components. When $A$ is the Neyman-Pearson likelihood ratio, the cryptic component vanishes, connecting our result to optimal e-value design. We demonstrate the theory on high-dimensional image data by empirically constructing this I-projection. By conditionally resampling EMNIST letters to identically match the conformity scores of MNIST digits, we demonstrate that a distribution can be shifted to a disjoint support while leaving the test martingale mathematically blinded.