When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets
Software EngineeringArtificial Intelligence
Summary
The authors studied how different software tuning methods work when you have more or less time to adjust settings for various systems. They tested eight optimisers on 22 systems and found that some work better when you have limited time, while others shine when you have more time. Notably, one method called FLASH did well across almost all systems regardless of time limits. They suggest this happens because many systems have good local solutions that greedy methods like FLASH can quickly find.
Authors
Chao Jiang, Yulong Ye, Tao Chen, Miqing Li
Abstract
Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade. Yet, the time required during the tuning process may vary across systems. In some systems (e.g., PostgreSQL), it may take a few minutes to measure a configuration, whereas in some others (e.g., MariaDB), it can take several hours. Moreover, even within the same system, users may have varying budgets and preferred settings. This naturally raises a question -- Given a budget level, which optimiser is the best choice for SE practitioners? This matters because optimisers usually have their own ``comfort zone'' and may perform very differently under distinct budgets. In this paper, we aim to answer this question. We systematically evaluate eight well-established optimisers across 22 configurable systems under varying budget levels. We find that, unsurprisingly, model-based optimisers (e.g., SMAC) are well-suited under tight budgets, and model-free optimisers (e.g., GAs) become superior with more generous budgets. However, interestingly, there is one optimiser, FLASH, that performs consistently well on most systems regardless of budgets. We lastly investigate the reasons behind this phenomenon and find that many systems possess good local optima (with large basins of attraction), allowing greedy optimisers (e.g., FLASH) to achieve strong performance. Source code, data, and supplementary materials of this work are available at https://anonymous.4open.science/r/Config-W2W-98B2.