Neural networks improve statistical tests with tricky unknown factors
Likelihood-free inference with nuisance parameters through normalizing flows
Machine Learning
Summary
Sometimes when analyzing data, scientists need to understand the effect of one thing while ignoring other unknown or confusing factors, called nuisance parameters. This paper describes a new way to use neural networks to create a special summary of the data that ignores these tricky factors but still helps make good decisions. The method works well with small to moderate amounts of data and can handle common patterns like shifts in scale or location. By doing this, it gives more reliable test results and runs faster than some traditional methods.
normalizing flowslikelihood-free inferencenuisance parameterspivotal statisticKL-divergencestatistical powergroup invarianceprofile likelihood-ratio testWelch testcalibration
Authors
Phil Assheton
Abstract
We present a simple decomposition of a neural-network-based normalizing flow that naturally uncovers a pivotal statistic (or something close) in the presence of nuisance parameters, based only on a sample generator from the distribution of interest. We show that the statistic is near-pivotal in the sense of minimum average KL-divergence of its $p$-values versus uniform and we argue that it can be expected to have good power when the dimension of the statistic equals the dimension of the parameter. It is able to incorporate prior knowledge about group invariances such as translation and scale. It can discover the one-sample $t$-test almost exactly, outperforms the Welch test in terms of worst-case size over a constrained variance-ratio range and achieves good calibration on partial biserial correlations, while showing higher power (and being much faster) on small-to-moderate samples than profile likelihood-ratio techniques.