Cost-aware stopping improves freeze-thaw Bayesian optimization utility
Cost-Aware Stopping for Freeze-Thaw Bayesian Optimization: When Does One-Step Lookahead Suffice?
Machine Learning
Summary
Freeze-thaw Bayesian optimization is a way to test options bit by bit, saving time by pausing and resuming tests. The authors study when a simple decision rule—stopping when the expected gain no longer beats the cost—is good enough. They find that while this rule usually avoids losing value, it might stop too soon, especially when costs are low. They propose a two-step check that can correct premature stops and show when each approach works best in practice.
What this means in practice
- •For machine learning engineers: Make tuning model hyperparameters more efficient by deciding when to stop incremental training evaluations based on cost versus expected gain.
- •For cloud service operators: Reduce compute costs by stopping incremental evaluations of candidate models early when expected improvements are outweighed by computation expenses.
Authors
Long Vu, Jongseong Brad Choi, Binoy Ravindran, Zhoulai Fu
Abstract
Freeze-thaw Bayesian optimization (BO) evaluates candidates in resumable increments but usually stops at a budget or convergence test. We maximize a budget-free utility, the best observed score minus a cost per increment, with a one-step lookahead: advance the candidate with the largest expected improvement (EI) under a fixed surrogate, and stop once no EI exceeds the cost. This rule, EI stopping, never falls below stopping immediately in expected utility, yet it can stop too early. We construct a certificate, a lower bound on the value of two-step lookahead, that proves such a stop premature, and repeating this correction never lowers expected utility under the surrogate. Under the surrogate, EI stops are premature on average at low costs and lose little at high costs. On three hyperparameter optimization benchmarks, a pre-registered test finds EI stopping noninferior to the correction in most high-cost settings, while the correction gains at low costs on two of the three; more information sharing in the surrogate raises EI stopping's utility. On the surrogates of two published methods, EI stopping trades lower scores for fewer training increments. EI stopping is a simple cost-aware reference whose utility depends on the surrogate.