Post incident software changes take longer and focus on server setup
Characterizing Post-Incident Software Evolution
Software Engineering
Summary
When something goes wrong in software, fixing it often involves a special kind of change called a post-incident update. The authors studied Wikimedia’s software over 10 years and found these fixes take more than twice as long as usual changes. They also tend to happen mainly in the parts of the system that manage server configuration. This new understanding helps software teams learn better ways to improve software reliability after problems.
What this means in practice
- •For software reliability engineers: Identify and allocate sufficient time and resources for post-incident fixes, especially focusing on server configuration changes.
- •For incident response teams: Adjust incident management workflows to accommodate the longer and more concentrated nature of post-incident software changes.
Tested on one dataset.
Authors
Matt Pope, Jonathan Sillito
Abstract
BACKGROUND: After a software incident, organizations attempt to improve the reliability of their software systems by conducting post-incident analysis and identifying actions to prevent or mitigate similar future incidents. Despite the importance of this activity, there is a limited understanding of the nature and characteristics of the software evolution triggered by the analysis process. AIMS: This study seeks to bridge this gap by providing the first empirical characterization of post-incident software evolution. We aim to establish an understanding of how these changes manifest, specifically focusing on how their characteristics (such as effort, location, and timing) differ from standard maintenance and evolutionary changes. METHOD: We conducted a mixed-methods analysis of a 10-year longitudinal dataset from Wikimedia, examining 1,302 tasks and 2,203 code reviews. We categorized these changes across five dimensions: target, effort, abandonment rate, prioritization, and time-to-resolve. RESULTS: Our results show that the post-incident evolution is distinct from other change types. Post-incident changes are more time-consuming, with a median completion time over twice that of other changes and are disproportionately concentrated in a server configuration repository. CONCLUSION: By identifying a few of the characteristics of post-incident evolution, this work provides a foundation for future research into reliability engineering and learning from failure. We highlight further avenues for exploration necessary to fully understand post-incident evolution. Lastly, our work enables researchers to move beyond high-level analysis and begin developing specialized methods and tooling tailored to the nature of post-incident evolution.