Benchmark tests large language models on reducing coding errors safely
LintEval: Benchmarking LLMs on Linter-Guided File Refactoring
Software EngineeringComputation and Language
Summary
Programs often have coding style or error warnings called lint alerts, which developers fix without changing how the program works. The authors created LintEval, a large test set to check if AI code models can fix these warnings while keeping the code working correctly. They found that some big commercial models reduce many alerts but sometimes break code functionality, while smaller models struggle to reduce alerts at all. They also showed that training models specifically on refactoring examples helps improve their performance significantly.
What this means in practice
- •For software developers: Evaluate and improve AI tools that automatically clean up code warnings without breaking functionality across multiple programming languages.
- •For software quality assurance teams: Test AI-based code refactoring assistants to ensure they reduce lint errors while preserving program correctness during code reviews.
Authors
Tongwei Deng, Jinyu Zhu, Yue Cao, Chenxu Wang, Yanzhen Zou, Bing Xie
Abstract
Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code benchmarks do not explicitly evaluate linter-guided file refactoring, where models must reduce lint alerts in source files with correct initial functionality while satisfying functionality preservation. We introduce LintEval, a multilingual benchmark for linter-guided file refactoring across C++, Python, and Java, comprising 13,061 programs with correct initial functionality and 171,662 lint alerts extracted using language-specific linters. LintEval jointly evaluates functionality preservation, lint alert reduction, new alert avoidance, and edit minimality. Experiments reveal a persistent trade-off: closed-source commercial models reach up to 90.4% Reduction@P but achieve only 67.1%-68.3% Pass@1, while several open-source small models show low or negative Reduction@P. A controlled fine-tuning study further shows that supervision format matters: refactoring-oriented SFT raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%. LintEval enables controlled evaluation of models that improve source files under functionality preservation.