Adaptive optimization method routes budget based on problem landscape
Adaptive Multi Mode: Budget-Aware Routing of Optimizers By Online Landscape Probes
Machine LearningNeural and Evolutionary Computing
Summary
Different algorithms work better or worse depending on how complex a problem is and how long you can spend solving it. The authors created a system called AMM that tests small pieces of the problem to guess its shape and decides how to split the work among several algorithms accordingly. This approach adjusts automatically during solving, leading to better results on standard test problems than any single algorithm alone. AMM is available as open-source software for others to use.
What this means in practice
- •For machine learning engineers: Optimize tuning hyperparameters efficiently by adapting method choice to problem structure and computation budget automatically.
- •For software developers: Incorporate AMM to improve performance of continuous optimization tasks in applications where problem characteristics vary widely.
Authors
Benjamin Jacob Bodner
Abstract
The best classical optimizer for a continuous black-box problem depends on its budget and landscape. Existing routers decide from the dimension and budget, past improvement or features fitted to one benchmark, and few test an optimizer's implicit assumptions online. AMM (Adaptive Multi Mode) routes a known budget among quasi-Newton, trust-region, coordinate, multistart and evolution-strategy optimization engines with online probes of curvature, plateaus, separability, kinks, basins and fractal structure. Each probe has a default, a dimension-by-budget validity range and a cost charged to the budget. In its pre-registered development loop on bbob, 32 of 36 predictions held over the last 16 confirmations. AMM was then tested once on held-out bbob instances 101-110 (D=2-40, 10*D-1000*D) against up to seven competitors. It has the highest target share in 382 of 600 function-dimension-budget cells (236 outright, 146 tied) and, by a descriptive posterior rule, is credibly behind in 35, mostly at 10*D and 30*D. At every budget it also leads the controls built from its own engines (the best engine alone, the best engine per dimension and budget, and a greedy schedule), narrowly at 10*D and 30*D, and nevergrad's NgIohTuned. In pre-registered runs its lead over IPOP-CMA-ES holds under affine and box transformations, on box-constrained bbob and, pooled, on rotated f2-f4, but rotating the separable f3 and f4 removes most of the lead there. On held-out MA-BBOB its final share at 2000*D is level with IPOP-CMA-ES's and its anytime area lower. We published AMM as an open-source Python package (pip install amm-optimizer; https://github.com/BjBodner/amm-optimizer).