Evaluation metrics predict test effectiveness for combinatorial testing
An Extensive Empirical Study on Evaluation Metrics for Combinatorial Interaction Testing
Software Engineering
Summary
Software testers use combinatorial interaction testing to find bugs caused by how different settings interact. But choosing the best way to measure the quality of these test sets is tricky because many options exist. The authors studied many such measurement methods across real software projects and found that some metrics predict bug finding better than others. They also show that some metrics are cheaper to calculate and that test set features affect these metrics' usefulness. They offer practical advice to help testers pick the right measurement for their needs.
What this means in practice
- •For software testers: Assess and compare combinatorial test suites using validated metrics to better predict their fault-detection ability.
- •For test automation engineers: Choose evaluation metrics that balance accuracy and computational cost to optimize automated test generation workflows.
Authors
Lisha Qin, Chenhui Cui, Tao Li, Rubing Huang, Shikai Guo, Lei Ma
Abstract
Combinatorial interaction testing (CIT) is a black-box testing method that has received extensive attention in both research and practice over recent years. Its primary objective is to construct an effective combinatorial test suite that detects software failures caused by parameter interactions. As a fundamental component of the CIT testing process, the evaluation metric plays a critical role in assessing and comparing combinatorial test suites, as well as in evaluating various test generation techniques. For CIT practitioners, selecting an appropriate evaluation metric is both important and challenging, given the wide variety of available options. Nevertheless, no prior work has systematically addressed this problem. To fill this gap, this paper first provides a comprehensive survey of black-box evaluation metrics for combinatorial test suites, offering rigorous definitions, clear classifications, illustrative examples, and complexity analyses. We then conduct an extensive empirical study involving eight open-source projects, encompassing 32 test scenarios and 295,624 combinatorial test suites. In this study, we examine the correlation between each static evaluation metric and fault-detection effectiveness using two correlation measures. Experimental results show that the Value Combination Coverage (VCC) metric serves as a valid predictor for test-suite evaluation. However, distribution-based metrics generally incur lower computational costs than interaction coverage-based ones. The choice of an appropriate metric should also account for the test suite's inherent properties, as these characteristics can substantially influence the effectiveness of the evaluation. Finally, we provide practical guidelines to assist CIT practitioners in selecting suitable evaluation metrics for assessing or comparing combinatorial test suites.