Vocabulary analysis detects flaky Swift tests with random forest model
The Vocabulary of Flaky Tests in Swift
Software Engineering
Summary
Tests that randomly fail without code changes cause delays and frustration in software development. The authors studied ways to predict such flaky tests in Swift code by looking at the words used in test code. By training a machine learning model on 91 flaky and over 22,000 stable tests, they found that certain keywords related to concurrency and timing help identify flaky tests. This approach works well but struggles when flakiness depends on parts of the system outside the test code itself.
What this means in practice
- •For mobile app developers: Identify flaky Swift tests early by analyzing test code vocabulary to improve continuous integration reliability.
- •For software quality assurance teams: Use vocabulary-based classifiers to prioritize test reviews and maintenance by detecting likely flaky tests in Swift projects.
Authors
João Medeiros, Denini Silva, Breno Miranda
Abstract
Flaky tests produce non-deterministic outcomes without code change, eroding CI confidence and delaying deliveries. While vocabulary-based machine learning prediction has proven effective for Java and JavaScript, no study has evaluated it for Swift, a language whose testing style is dominated by UI and asynchronous code. We collect 91 flaky and 22,349 stable tests from 15 open-source Swift projects via re-execution and commit-history mining, then train five classifiers (Random Forest, Decision Tree, Naive Bayes, SVM, KNN) on TF-IDF unigram+bigram features under stratified 5-fold cross-validation. Random Forest achieves the best performance (Precision = 0.92, F1 = 0.86, AUC = 0.95) and substantially outperforms trivial baselines, among them a vocabulary-threshold rule applied to the most informative tokens, confirming a genuine discriminative signal (MCC = 0.75 vs. 0.08 for the best baseline). Information-gain analysis reveals two complementary signal types. Flakiness markers appear predominantly in unstable tests and comprise concurrency primitives (async, await), expectation-based synchronisation (expectation, fulfill), error propagation (throws), and explicit timing dependence (timeout, wait, now). Stability markers, chiefly the assertion vocabulary of plainly synchronous tests (xctassertequal), count as evidence against flakiness. Error analysis shows that the model fails when flakiness is hidden in shared infrastructure outside the test body or when async constructs are used in a deterministic context, exposing the intrinsic limit of lexical prediction. These results extend vocabulary-based flakiness detection to the Swift ecosystem and characterise both its effectiveness and its boundaries.