Three-Phase Evaluation of AI-Assisted Software Development Life Cycle
2026-07-06 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors studied how letting AI take over more of the software development process affects programmers' work. They had four developers build the same web app three ways: using partial AI help, fully relying on one AI tool, and then another AI tool. They found that more AI control cut down work time, made the final product match requirements better, and lowered how hard developers felt they were working, though frustration slightly rose. The second AI tool seemed to work best, indicating the type of AI system matters, not just how much control it has.
AI autonomysoftware development productivityrequirement adherencecognitive workloadGitHub CopilotAWS KiroNASA-TLXRITM scoreAI-assisted developmentfull-stack web application
Authors
Joshua Strubel, Professor Carrie Russell, Carson Crockett, Jason Ferraro, Nathan Londhe, Uzayr Syed, Jacob Viehe
Abstract
This paper presents an exploratory evaluation of how increasing levels of AI autonomy affect software development productivity, requirement adherence, and developer cognitive workload. A team of four developers reimplemented the same full-stack web application across three sequential phases: partial AI-assisted development using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro. Evaluation metrics included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures. Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload, while developer frustration increased modestly. The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.