Developers rarely test code marked with self-admitted technical debt
Is Self-Admitted Technical Debt Tested? An Empirical Study of Coverage, Co-change, and Impact
Software Engineering
Summary
Sometimes developers leave notes in their code saying it’s not perfect and needs fixing later. This paper studied whether those notes, called self-admitted technical debt, get extra testing when fixed. The authors found that about 60% of this code is covered by tests, but when the code is improved, developers seldom add or update tests specifically for the fix. Over time, fixing these parts with or without new tests leads to similar rates of bugs, suggesting developers treat these fixes like normal changes rather than risky ones needing special checks.
What this means in practice
- •For software engineering managers: Identify gaps in testing practices during technical debt cleanup to improve code quality assurance processes in development teams.
- •For software quality assurance teams: Guide focus on adding targeted tests when resolving self-admitted technical debt to potentially reduce future defects.
Authors
Suzuka Yoshimoto, Kosei Horikawa, Daniel Feitosa, Yutaro Kashiwa, Hajimu Iida
Abstract
When developers write a TODO or FIXME comment, they are explicitly admitting that the code is suboptimal: a built-in warning that this logic deserves extra scrutiny. Yet it is an open question whether Self-Admitted Technical Debt (SATD) actually receives that scrutiny in the form of software testing. We aim to characterize the relationship between SATD and testing across three dimensions: the extent to which SATD-affected code is covered by existing tests, whether developers synchronize test additions with debt resolution, and whether such testing affects the long-term observability of resulting defects. For that, we conducted an empirical study on eight open-source Java projects, analyzing test coverage of 784 SATD instances identified in the latest releases and performing a longitudinal examination of 5,175 SATD removal events. Our results show that while 60.7% of SATD-affected code is covered by existing test suites, developers rarely synchronize test modifications with debt resolution; manual inspection confirms that only 3.4% of SATD removal commits include new tests specifically targeting the resolved debt (vs. 12.5% that co-add tests in the same commit). Longitudinal analysis further suggests that SATD resolutions exhibit nearly identical localized bug induction rates within short-to-medium-term windows regardless of test modifications. However, over a longer, unrestricted observation window, a slight divergence emerges where the test-added group reaches a higher cumulative defect alignment probability (6.32% vs. 4.37%), a counterintuitive trend potentially driven by the selective testing of inherently complex components. Developers treat SATD repayment as an ordinary code change rather than as a high-risk maintenance activity: most debt removals proceed without targeted verification, despite the developer's own prior flag that the code is suboptimal.