Learning-based mutant selection risks missing code faults without line coverage
Where Does the Budget Go? Structural Concentration in Learning-Based Mutant Selection
Software Engineering
Summary
Mutation testing checks software by making small changes called mutants to find bugs. The authors found that learning methods to pick the most useful mutants often focus too much on a few parts of the code, leaving other areas untested and allowing bugs there to go unnoticed. They introduce LinePool, a simple way to spread out mutant selection across different code lines to cover more ground. This method keeps the good ranking of mutants while testing more areas and finding more bugs, especially when testing resources are limited.
What this means in practice
- •For software testing teams: Improve test effectiveness by using LinePool to select mutants across more code areas, reducing missed bugs within limited testing budgets.
- •For quality assurance engineers: Make mutant selection more robust and diverse to better detect faults even when ranking predictions are imperfect.
Authors
Taha Rostami, Jeongju Sohn, Mike Papadakis
Abstract
Learning-based mutant selection is used to reduce the cost of mutation testing by ranking and selecting the mutants that seem to be the most promising. Prior work has shown great potential for this approach in fault revelation and subsuming-mutant selection. However, the risks and benefits of these approaches -- especially the distribution of selected mutants across code locations and how that distribution affects behavioral diversity and fault revelation -- remain unclear. We investigate this distribution and its effects, and introduce LinePool, a simple, model-agnostic stratification step that distributes selections across source lines while retaining the underlying ranking signal. Across two datasets and budgets (2% and 5% of the mutants), we find that learning-based selection concentrates the budget on a few code locations, and may thereby systematically introduce 'blind spots', untested code areas, allowing faults to escape detection. On larger programs, the selected mutants also exhibit substantial overlap in kill behavior. LinePool substantially increases spatial coverage and reduces kill-set overlap. It also improves fault revelation at tight budgets and makes selection more robust to imperfect or shifted ranking signals. Comparisons with clustering and established diversification methods show that none consistently outperforms LinePool across datasets, supporting its use as a simple diversification step. In general, our results suggest that behavioral diversity should be considered alongside effectiveness when designing and evaluating mutant-selection methods, and that exploiting structural elements of the program is one simple way to obtain such diversity while maintaining effectiveness.