Open-weight language models label bug-fixing commits better than keywords
Labelling Bug-Fixing Commits with Local Open-Weight Language Models
Software Engineering
Summary
Identifying which changes in software fix bugs is important but often done with noisy or biased methods like using issue trackers or keyword searches. The authors tested local, open-weight language models that look only at the commit messages and code changes to automatically identify bug fixes. Their method found many more true bug-fixing commits than simple keyword searches across several programming languages. They also provide tools and datasets to support building better defect prediction systems.
What this means in practice
- •For software development teams: Automatically mark commits that fix bugs using only their content without relying on issue trackers or manual labelling.
- •For software quality engineers: Build and maintain cleaner, up-to-date datasets of bug fixes for defect prediction and quality analysis in multi-language codebases.
Authors
Philip König, Georg Goldenits, Caroline König, Sebastian Raubitzek, Fabian Obermann, Dennis Toth, David Schmidt, Edgar Weippl, Kevin Mallinger
Abstract
Defect prediction depends on knowing which commits fix bugs, yet the labels that encode this are produced by routes that each introduce noise. Reused benchmarks carry documented data-quality problems, issue-tracker links are biased and the underlying reports are frequently mistyped, and matching keywords in commit messages is a coarse heuristic. This paper examines whether commits can be labelled as bug fixes from their content alone, using open-weight language models that run locally and therefore keep the process reproducible, inexpensive at corpus scale, usable on proprietary code, and independent of any issue tracker. Against datasets of manually validated and curated bug fixes spanning Java, Python, and JavaScript, we compare a keyword baseline with a set of open-weight models of varying size, prompting each with the commit message and the code diff. On the manually validated corpus the keyword baseline recovers fewer than half of the fixes, whereas the open-weight models recover the large majority and outperform it repository by repository with statistical significance, and larger models do not consistently outperform smaller ones. We further show that evaluation corpora without negative examples cannot support a precision-aware comparison of such classifiers. We release the labelling pipeline together with a labelled, multi-language corpus produced by the recommended configuration, as a reproducible silver-standard resource for building current, project-specific datasets.