Software engineering expertise enhances coding large language models
Software Engineering LLMs: Why We Need Them and How Software Engineering Helps Build Them
Software EngineeringMachine Learning
Summary
Coding large language models (LLMs) can be improved by using knowledge from software engineering. The authors show how software engineering methods help choose what skills these models should learn, create trustworthy training examples, and measure progress. They developed a tool called RefactorForge to make training data for complex code changes, leading to better performance on coding tasks. The study suggests combining software engineering and AI training can make coding models more useful and reliable.
What this means in practice
- •For software development teams: Train coding AI assistants to perform reliable large-scale code refactoring based on verified training data from real projects.
- •For devops engineers: Improve automated issue resolution by incorporating software engineering knowledge into language models used in continuous integration pipelines.
Authors
Amir M. Ebrahimi, Mohammed Mehedi Hasan, Seyed Shayan Daneshvar, Jiahuei Lin, Gopi Krishnan Rajbahadur, Ahmed E. Hassan
Abstract
Software engineering (SE) expertise can shape how coding large language models (LLMs) are trained and which engineering capabilities they acquire. We examine this role through an industrial case study of capability-focused training for a 92B-parameter coding LLM serving more than 100,000 developers. We demonstrate how established SE knowledge can guide the selection of engineering capabilities, the construction of verifiable training data, and the evaluation of model improvements. To target repository-level refactoring, we developed RefactorForge, a pipeline that combines refactoring knowledge, established SE tools, and verification techniques to construct training data. Across 306 Python repositories, RefactorForge generated 5,704 executable tasks and retained 4,572 verified teacher trajectories totaling 72.51 million tokens. Incremental supervised fine-tuning on these trajectories, accounting for 0.46% of the preceding coding mixture's tokens, improved RefactorBench performance by 18.00 percentage points and SWE-Bench ProMax performance by 7.06 points. Under OpenCode, these improvements extended to SWE issue resolution, with gains of 4.97 points on SWE-Bench Verified and 1.23 points on SWE-Bench Pro. Improvements also persisted across other coding-agent harnesses, while greenfield repository generation showed mixed results. We report lessons from constructing capability-focused training data and evaluating its effects across engineering workloads. Building on this industrial experience, we propose a research agenda for expanding SE's role in transforming coding LLMs into SE LLMs: incorporating SE knowledge into model training, applying engineering methods to model development and maintenance, and systematically defining the engineering capabilities these models should acquire.