Shaker shows no advantage over rerunning flaky tests in python projects
Evaluating Shaker for Flaky Test Detection in Python Projects
Software Engineering
Summary
Flaky tests sometimes pass and sometimes fail even when nothing changes, which causes confusion and extra work for developers. The authors looked at a tool called Shaker, which was good at finding flaky tests in Java but did not work better than simply running tests multiple times in Python projects. They found that many flaky tests don’t show flaky behavior consistently on different computers, and when they do, it’s mostly due to networking or randomness, not the concurrency problems Shaker tries to detect. This means tools evaluated on flaky tests from other setups might seem less effective than they really are.
What this means in practice
- •For python software developers: Choose simple repeated test runs over resource contention injection when detecting flaky tests in Python projects.
- •For software testing teams: Recognize that flaky test detection tools tested in one environment might underperform when used in different execution settings.
Authors
Gabriela Leal, Denini Silva, Leopoldo Teixeira
Abstract
Flaky tests pass or fail non-deterministically on unchanged code, eroding trust in test suites and inflating the cost of every failure. Shaker detects them by injecting resource contention (CPU, memory, and I/O stress) to amplify non-determinism caused by concurrent execution, and was reported to detect 95% of the flaky tests in a Java and Android benchmark against 37.5% for plain re-execution (ReRun). We present the first empirical evaluation of Shaker for Python. Drawing non-order-dependent flaky tests from the ground-truth dataset of Gruber et al., we compare Shaker against a budget-matched ReRun baseline in a paired design, giving both techniques the same number of test executions: Each of 137 tests is run 100 times under each. As configured for Java and Android, Shaker provides no statistically significant detection advantage over plain re-execution (37.2% vs. 35.8%; McNemar exact p = 0.84). Two findings explain why. First, fewer than half of the ground-truth flaky tests reproduce as flaky at all on independent hardware under either technique, and most of the tests that fail to reproduce never diverge once across 100 runs. Second, the tests that do reproduce are dominated by flakiness from network interactions and randomness rather than the concurrency Shaker targets. Beyond the tool, this exposes a broader hazard for the field: reusing a flaky-test ground truth across execution environments silently converts genuine flaky tests into apparent true negatives, deflating any tool's measured recall.