Software methods compared to spot performance drops early
Evaluating Change Point Detection Methods for Software Performance Regression Analysis
Software Engineering
Summary
Keeping software fast and reliable is very important for users and businesses. The authors studied how well different techniques can automatically find moments when software slows down or behaves worse, by looking at data collected over time. They gathered real performance data from big software projects and checked how people spot these slowdowns by hand. Then, they tested twelve different methods to see which ones do the best job at finding these changes. This helps people pick the right tools to catch problems early in software development.
What this means in practice
- •For software development teams: Identify automatic methods that reliably detect software performance drops during development cycles using real performance data.
- •For it operations teams: Improve monitoring systems by selecting the most effective change detection methods to catch performance degradations in running software.
Authors
Diego Elias Costa, Michele Tucci, Luca Traini, Daniele Di Pompeo, Thomas Bach, Francois Farquet, David Daly, Simon Eismann, Petr Tůma, Vittorio Cortellessa, André van Hoorn
Abstract
Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.