Fault localization helps but does not guarantee better AI bug fixes
From Finding to Fixing: How Useful is Fault Localization for Agentic Program Repair?
Software Engineering
Summary
Finding the exact spots in code that cause bugs is important for fixing software. This study looks at how useful spotting these fault locations is when AI methods try to fix code by themselves. The researchers found that knowing where the bugs are can help AI fix code more efficiently, but just being better at locating faults doesn't always mean more bugs get fixed. Also, giving AI explanations about the cause of the bug helps even more than just showing the spot. This means fault localization is helpful but has limits in improving AI-driven software repair.
What this means in practice
- •For software development teams: Improve AI-assisted debugging workflows by incorporating fault localization to guide early search for bug fixes and reduce repair effort.
- •For software testing specialists: Use fault localization combined with bug cause details to increase success rates in automated program repair tools.
Authors
Xinyu Shi, Zhou Yang, An Ran Chen
Abstract
Fault localization (FL) aims to identify code locations responsible for software failures and is commonly evaluated by how accurately it ranks known fault locations. Large language model (LLM) agents are changing program repair and are increasingly used to search and reason over repositories. Although recent LLM-based FL methods have improved localization accuracy, their impact on agentic repair remains unclear. To address this gap, we conduct a systematic empirical study of FL in agentic repair. We evaluate six recent FL methods on SWE-bench Verified and Defects4J with three LLMs under a common repair framework and analyze 16,248 repair trajectories. We also examine gold locations and bug-cause descriptions as forms of FL guidance across different repair contexts. The studied FL methods do not consistently improve repair success, and higher localization accuracy does not consistently lead to larger repair gains. Yet FL guides agents' early search and helps them reach relevant code sooner. Accurate locations can reduce repair token use by up to 39.2\%, but current FL methods often consume more tokens than they save during repair. Experiments with gold guidance suggest that FL has greater potential when bug inputs contain fewer location clues. Providing bug causes can further improve repair success beyond locations alone. We also find that ground truth from a single reference patch does not fully capture the locations used by successful repairs. These findings help clarify the value of FL in agentic repair and inform future work on its design and evaluation.