Fixed size sampling improves variance without extra uncertainty
Fixed-Size Active Statistical Inference
Machine Learning
Summary
When gathering data to make decisions, choosing how many samples to collect can be tricky because sometimes the number of samples is random and causes extra uncertainty. The authors looked at a common sampling method called Poisson Sampling, which picks a random number of samples, and showed that another method called Conditional Poisson Sampling can pick a fixed number of samples instead. This fixed-size method has as good or better accuracy than the usual approach, especially when predictions used in the process aren’t perfect. They tested this on real and simulated data and provided code for others to use.
What this means in practice
- •For data science teams: Use fixed-size sampling to improve reliability and reduce extra variance when selecting samples for large-scale data analyses.
- •For survey methodologists: Implement conditional Poisson sampling to ensure fixed survey sample sizes while maintaining or improving variance properties.
Authors
Erik Skalnes, Michael Oberst
Abstract
Variance reduction techniques improve the efficiency of drawing conclusions from data. Recent proposals for variance-reduction use two complementary techniques to reduce variance: incorporating machine learning predictions into estimators and weighted sampling. A popular weighted sampling algorithm is Poisson Sampling (PS), which independently flips a biased coin at each unit and includes all heads in the sample. A fundamental flaw with PS is that it has a random sample size, which presents practical challenges and introduces variance into the estimator. We show that a fixed-size alternative, known in survey sampling as Conditional Poisson Sampling (CPS), is a drop-in replacement for PS on general multivariate Z-estimation problems, which include convex M-estimation considered in prior work. Notably, we show that the asymptotic variance of CPS is no worse than that of PS, and can be substantially better in realistic scenarios where predictions are imperfect. We illustrate these advantages on both synthetic and real datasets, and release a Python package implementing CPS. In light of our contributions, there is no need to suffer a variable sample size.