Ai writes and tests 21000 lines of Python code with errors tracked
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
Software EngineeringArtificial Intelligence
Summary
This paper looks at a software project where an AI called Claude wrote all the Python code and tests, with no humans writing code. The authors studied the types of instructions given to the AI and how the code evolved over time. They found that about 14% of the AI-generated code had real errors that the AI’s own tests later caught, and about one in five interactive responses from the AI had factual mistakes. Their tools and analysis help us understand how reliable AI-written code and communication can be.
What this means in practice
- •For software development teams: Track and analyze the reliability and errors in AI-generated software projects by tracing code provenance and instruction types.
- •For automated testing engineers: Improve detection of errors in AI-authored codebases using AI-generated test suites and error analysis tools presented.
Authors
Douglas Leith
Abstract
We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.