IssueLighter improves detecting invalid open source software issue reports
IssueLighter: An Issue Type-Aware LLM Agent for Issue Report Validity Assessment
Software EngineeringArtificial Intelligence
Summary
Open source software projects get many issue reports, but many are invalid and waste maintainers’ time. The authors studied 520 reports to find why some are invalid across different issue types. They created IssueLighter, a system that understands issue types and the reasons reports can be invalid, improving detection and explaining why a report is invalid. Their tool was better than existing methods and even persuaded maintainers to revisit some decisions.
What this means in practice
- •For open source project maintainers: Automatically filter and explain invalid issue reports across various categories to reduce maintainer workload and improve triage efficiency.
- •For software development teams: Integrate type-aware issue validity checks to improve the quality of reported issues and streamline bug tracking processes.
Authors
Jiawei Ran, Yuxia Zhang, Ziyang Zhao, Hui Liu
Abstract
Invalid issue reports are prevalent in open source software (OSS) projects, imposing substantial burdens on maintainers who already face a high risk of burnout. Prior research has primarily focused on identifying invalid reports for specific issue categories (e.g., bug and wontfix), leaving the broader landscape of OSS issue types largely unexplored and limiting the generalizability of existing approaches. Moreover, existing methods provide little support for explaining or validating their predictions, reducing their practical usefulness for maintainers. With the rise of LLM-based agents, it remains unclear whether a comprehensive understanding of the causes of invalidity across all issue types can improve validity assessment performance while producing well-grounded rationales. To this end, we first manually annotated the validity status of 520 issue reports from three OSS projects, i.e., pandas, scikit-learn, and mypy, resulting in 257 invalid reports and 263 valid ones. Then, we conducted a thematic analysis on these invalid reports and identified 13 causes of invalidity that are closely specific to issue types. Based on these insights, we propose IssueLighter, an agent-based framework with a type-aware validity assessment strategy that dynamically executes commands. Experimental results show that IssueLighter consistently and substantially outperforms all baseline methods, with an average F1 improvement of over 20 percentage points. Ablation studies confirm that both the type-aware strategy and the taxonomy of causes of invalidity contribute to the performance gains. Qualitative analysis further demonstrates that IssueLighter generates rationales grounded in accurately identified causes of invalidity and supported by evidence. Moreover, leveraging these rationales, we challenged maintainers' original decisions in four disputed cases, and all received approval.