Ai-generated code is smaller and stylistically different than human code
What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code
Software EngineeringArtificial Intelligence
Summary
Many people use AI assistants to write computer code, but we don't fully know if AI code is as good as human code beyond just working correctly. The authors compared hundreds of thousands of functions written by humans and generated by AI in popular programming languages. They found AI code tends to be smaller and more repetitive, while human code has different kinds of bugs and security issues. The authors also created a benchmark to help test code quality and security more thoroughly.
What this means in practice
- •For software development teams: Use the CQBench benchmark to better assess and compare human and AI-generated code quality and vulnerabilities across common programming languages.
- •For security testing teams: Integrate CQBench's evaluation pipeline to detect and analyze security weaknesses specific to AI-generated code in multi-language environments.
Authors
Cristina Improta, Pietro Liguori, Domenico Cotroneo
Abstract
AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whether their code differs from human code in the quality dimensions dominating lifecycle cost. We compare human-written and AI-generated code at scale: 787,562 function pairs across Python, Java, and C, each human function mined from open-source repositories paired with implementations generated from its docstring by three AI assistants (OpenAI GPT models, DeepSeek-Coder, Qwen2.5-Coder). We characterize structural complexity and statistical naturalness, and map static-analysis findings onto Orthogonal Defect Classification for defects and the Common Weakness Enumeration for vulnerabilities, making authors and languages directly comparable. AI-generated code is structurally compressed and stylistically templated: roughly half the size and branching of human code, clustering apart at the style level. Defect profiles differ in kind: human code concentrates issues of mature codebases, AI code repetitive boilerplate; security is language-dependent, with LLMs producing more, and more severe, findings in Python and Java but fewer high-severity memory-safety findings than humans in C. Once size is controlled for, complexity metrics carry little signal, while naturalness separates authors. Finally, we release CQBench, a benchmark of 27,346 issue-prone tasks with baselines and an evaluation pipeline for quality assurance and security testing.