Llms improve repairing broken tests after code changes effectively
Finding the Pivots to Repair Broken Source-Test Evolution Using LLMs
Software Engineering
Summary
Software tests often break when the programs they check get updated, making it hard to keep tests reliable. The authors studied how large language models (LLMs) can help fix these tests by understanding code changes and error messages. They created a large collection of real examples from popular Java projects to test different ways LLMs can repair tests. Their work shows combining various information sources and letting LLMs revise their fixes improves success, and they developed a guide to help LLMs reason better while repairing tests. These improvements made an existing coding assistant much better at fixing difficult test problems.
What this means in practice
- •For software developers: Fix broken tests automatically after code changes to maintain test reliability using the improved LLM-based repair techniques described.
- •For continuous integration engineers: Integrate LLM-guided test repair into CI pipelines to reduce manual test maintenance efforts and speed up development workflows.
- •For qa automation teams: Use enhanced reasoning-driven LLM agents to automatically update tests that fail due to source code evolution, improving software quality assurance.$Commercial implications: Enables building smarter test automation tools that automatically fix obsolete tests, a marketable upgrade for QA software providers.
Authors
Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, Shing-Chi Cheung
Abstract
Repairing obsolete tests in response to source code evolution is critical for test suite reliability and software quality assurance. While large language models (LLMs) have emerged as promising tools for automating many software development tasks, the effective supply of context and the orchestration of repair pipelines in this source--test co-evolution scenario remain poorly understood. Moreover, although reasoning is found helpful for LLMs on some complex coding tasks, it remains unclear which reasoning steps actually contribute to effective test repairs and how the insights learned from LLMs' behaviors benefit coding agents. In this paper, we present a systematic empirical study of test repair for source-test co-evolution empowered by LLMs, investigating the effective setups and typical LLM behaviors. To support this study, we construct a new benchmark of 500 real-world source-test co-evolution cases from commits submitted between May 2024 and April 2026 across 15 highly-starred open-source Java projects. Our experimental results reveal several key insights for effective LLM-based test repair: the complementary benefits of combining test failure symptoms, source code changes, and supportive code context, as well as the effectiveness of self-repair rounds. We further derive a common reasoning action taxonomy from LLMs' reasoning traces, which proves instrumental in guiding LLMs to repair tests more effectively while at a lower financial cost. Applying our findings to the Codex agent further improves its repair success rate on challenging cases by 66.7%.