Large language models produce varying quality code beyond correctness
Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code
Software Engineering
Summary
Checking whether computer code works correctly is important, but it doesn't tell the whole story. The authors studied code written by AI models and found that some produce code that is harder to maintain or less reliable, even if it passes basic tests. They used tools that look at how the code is structured and how easy it is to update or fix. Their findings suggest that judging AI code by whether it runs correctly misses important differences in quality.
What this means in practice
- •For software development teams: Improve code review processes by including static quality metrics for AI-generated code to ensure maintainability and reliability.
- •For software quality assurance teams: Design testing frameworks that assess AI-generated code beyond just passing tests, incorporating code structure and quality measures.
Authors
Jue Huang, Tarek Mahmud, Meiru Che, Guowei Yang
Abstract
Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that code is maintainable, reliable, or well-structured. This paper {presents a large-scale static quality profiling of} LLM-generated Python code, with a focus on maintainability and reliability. Using 65{,}663 programs from 73 models in LiveCodeBench, we evaluate generated code with established static analysis tools, including Pylint, Radon, and SonarQube. Our results show systematic differences across models that are not captured by correctness alone, with maintainability emerging as the clearest source of variation. We further find that correctness often comes with structural trade-offs rather than uniformly simpler solutions. Overall, our findings motivate evaluation frameworks that assess LLM-generated code beyond pass rates alone.