AdapBench reduces microbenchmark time up to 85 percent in cloud CI CD
AdapBench: Adaptive Early-Stopping and Prioritization for Efficient Microbenchmarking in Cloud-based CI/CD
Software EngineeringDistributed, Parallel, and Cluster ComputingPerformance
Summary
Software updates released in the cloud often slow down user programs, costing money and energy. The authors created AdapBench, a tool that runs small tests to catch these slowdowns more quickly. It decides which tests are most important and stops running tests early once it has enough proof. In their tests on popular Java and Go projects, AdapBench cut testing time by up to 85% without missing real slowdowns or raising false alarms.
What this means in practice
- •For cloud software teams: Quickly detect software performance issues during continuous integration with shorter test runs and fewer false alarms using AdapBench's dynamic test scheduling.
- •For performance engineers: Reduce the time and cost of running microbenchmark suites by prioritizing informative tests and stopping redundant ones early in cloud-based projects.
Authors
Sebastian Koch, Jonas Heisterberg, Nils Japke, Trever Schirmer, David Bermbach
Abstract
Cloud-based software systems frequently release new versions, a fact further accelerated through agentic coding. Each new release, however, brings the risk of performance regressions which can irritate end users, increase infrastructure costs, and increase carbon footprints. Therefore, developers strive to catch such regressions in testing stages, usually through so-called continuous benchmarking. The standard way of doing this is using microbenchmark suites which, however, take notoriously long to execute. Optimizing the execution of microbenchmark suites is, hence, an active field of research. In this paper, we propose AdapBench, an online scheduling system for reducing the execution duration of microbenchmark suites. AdapBench is the first approach which dynamically makes at-runtime decisions for prioritizing the most informative benchmarks and early-stopping each one once enough evidence has accumulated to accept or reject a performance change hypothesis. We evaluate our prototype on eight popular Java projects, commonly used by microbenchmark research, and four additional Go projects. At matched 90% change detections and false-positive rate compared to an optimistic uniform run, AdapBench saves 42% of the aggregate execution duration on the Java projects, with negligible scheduling overhead (below 0.001% of the total runtime). On the Go projects, AdapBench finds every performance change with zero false positives while saving 85% of the full run.