Testing equivalence in machine learning methods with multiple runs
Equivalence testing and run-count design for stochastic machine-learning experiments
Machine Learning
Summary
Machine learning experiments often compare methods by running each just a few times, but this can be misleading because results vary each run. The authors created a method to test if two training approaches are truly similarly effective by modeling variability in two parts: differences between runs and differences within the data items. Their technique uses statistical tests to confidently say if methods perform equivalently within a margin, and guides how many runs are needed to make that claim reliably. They show their approach works on real-world tasks, where usual practice with a few runs leaves results uncertain.
What this means in practice
- •For machine learning engineers: Determine the number of training runs required to confidently show two algorithms perform equally on classification tasks.
- •For software qa teams: Design tests to certify that changes to model training do not degrade performance beyond a small, defined margin.
Authors
Dmytro Knopov, Nadiya Shvai
Abstract
Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (TOST), equivalently by a confidence interval inside the margin; a two-level bootstrap covers composite statistics. A closed-form formula converts a margin and a target power into runs per method; its shifted-t refinement is calibrated against the realised two-level process. Simulations on a 270-cell grid and under non-Gaussian seed laws confirm the formula within its design-relevant range, from five runs per method. For four published runs of a joint energy-based model, the failure to certify equivalence was largely decided by design: with the variance calibrated to those runs, two per method certify with probability near three per cent, whereas certifying 0.01 AUROC at 80 per cent power needs twelve even under exact equality. A second case study on a thousand-seed CIFAR-10 zoo and the MultiBERTs checks the formula against real seed laws (within 0.05 in that range) and resolves comparisons that two or three runs leave undecided. In a two-stage design (pilot, then formula), our experiments suggest that an upper confidence bound on the pilot variance substantially reduces the undersizing seen with its point estimate.