DisCoMBO improves optimization by combining models and expert knowledge
DisCoMBO: Steering Expert-in-the-Loop Black Box Optimization via Distributional Conformance
Machine Learning
Summary
Optimizing solutions often involves guessing where to look next based on uncertain information. Traditional methods rely on fully data-driven models, but these can ignore valuable expert knowledge. The authors introduce a new way to measure how surprising or uncertain the model's predictions are and use that to better incorporate expert insights while searching for solutions. Their approach also guarantees good long-term performance and works well in tasks like automated machine learning and wind park design.
What this means in practice
- •For machine learning engineers: Enhance automated model tuning by integrating expert knowledge with efficient uncertainty estimates leading to better hyperparameter optimization.
- •For energy system planners: Improve wind park layout optimization by combining learned models with domain insights to accelerate finding effective configurations.
Authors
Jonas Seng, Bennet Wittelsbach, Kristian Kersting
Abstract
Sequential Model-Based Optimization (SMBO) traditionally relies on Bayesian or ensembling surrogates for uncertainty quantification. While historically treated as fully data-driven, SMBO increasingly integrates external domain expertise to accelerate discovery. To overcome the opaque guidance and diminished integration fidelity of standard acquisition re-weighting, Probabilistic Circuits (PCs) have emerged as a generative surrogate alternative, enabling direct knowledge injection via conditional sampling. However, these generative routines lack the formal exploration-exploitation semantics required for rigorous optimization. We introduce the Distributional Conformance Score (DisCo), a novel metric that unifies the flexibility and efficiency of PCs with a formal uncertainty framework. DisCo provides a bounded, $[0, 1]$-normalized measure of model "surprise" that (1) recovers properties comparable to kernel-based uncertainty known from, e.g., Gaussian Processes, while maintaining linear-time inference, and (2) enables accurate assessment of conformance of external knowledge w.r.t. model evidence. We then present DisCoMBO, a framework leveraging these properties for robust, knowledge-aware optimization. We prove that DisCoMBO is a zero-regret algorithm and demonstrate its effectiveness across diverse benchmarks from AutoML, material optimization, and wind park optimization.