Improving black-box optimization with adaptive partitioning and neural models
Adaptive Partitioning Schemes for Optimistic Optimization
Machine Learning
Summary
Some problems need finding the best setting without knowing how the system works inside or having clues like slopes. The authors created a smarter way to split the problem into parts that fit the problem better and use flexible neural network helpers. Their method works better than previous ones when the main factors come from a smaller dimension inside a larger space. This helps improve quality in areas like compressing AI model weights with less accuracy loss.
What this means in practice
- •For machine learning engineers: Enhance neural network compression by improving quantization accuracy using adaptive optimization techniques demonstrated on large language models.$Commercial implications: Enables improved AI model compression products targeting companies deploying large-scale language models with better performance.
- •For engineering design teams: Optimize complex systems without known equations or gradients more efficiently by adapting domain partitions dynamically during optimization.
Authors
Raja Sunkara, Ardhendu Tripathy
Abstract
Applications such as engineering design often require us to optimize a black-box function, i.e., a system whose inner processing is not analytically known and whose gradients are not available. Practitioners often have a fixed budget for the number of function evaluations and the performance of an optimization algorithm is measured by its simple regret. In this paper, we study the class of "Optimistic Optimization" algorithms for black-box optimization that use a partitioning scheme for the domain. We develop algorithms that learn a good partitioning scheme and use flexible surrogate models such as neural networks in the optimization procedure. For multi-index functions on an $m$-dimensional subspace within $d$ dimensions, our algorithm attains $\tilde{O}(n^{-β/ d})$ regret, where $β= 1 + \frac{d-m}{2m-1}$, as opposed to $\tilde{O}(n^{-1/d})$ for SequOOL, a state-of-the-art optimistic optimization algorithm. We use our approach to improve the quality of Activation-aware Weight Quantization (AWQ) of the OPT-1.3B model, achieving $\sim10\%$ improvement in performance relative to the best possible unquantized model.