New ways to check engineering skills when AI helps build software
When AI Can Build It, What Should We Assess? Rethinking Evidence of Engineering Competence
Software EngineeringHuman-Computer Interaction
Summary
When AI can write code and create software parts for you, it's harder to tell how good someone really is at engineering by just looking at the final product. The authors redesigned a 13-week software course where students can use AI to do much of the writing, but must still do the planning and testing themselves. They created a system to check many different kinds of evidence, like how projects were made, tested, and explained, to better judge engineering skill. The study showed that relying on multiple sources and automated collection helps reveal true responsibility and capability.
What this means in practice
- •For software engineering instructors: Design course assessments that combine multiple evidence types to fairly evaluate student skills when AI produces code and documentation.
- •For software development team leads: Use structured processes to oversee AI-assisted development work, ensuring human responsibility and verification of AI outputs in projects.
Authors
Ali Shakiba
Abstract
Generative AI can produce code, tests, documentation, and technical explanations that resemble artefacts traditionally used as evidence of student competence, weakening the inference from artefact quality to the engineering capability behind it. We report the redesign and first deployment of a 13-week, AI-permissive, software-intensive engineering course in which students may delegate substantial work to AI while retaining responsibility for planning, verification, review, integration, and explanation. The redesign couples an engineering accountability workflow, which structures human-AI work and preserves human decision authority, with an assessment evidence architecture comprising six complementary channels: product, process/origin, verification, explanation, reflection, and individual assurance. A collect--cross-check--escalate (CCE) reasoning logic connects these channels rather than treating any single artefact or trace as sufficient evidence of competence. By the end of Week 11, the deployment involved 87 students organised into 17 teams, with 957 individual weekly Git repositories and 34 group-project Git repositories supporting repeated assessment and engineering activity. The first deployment yielded four practical lessons: AI assistance changes where rubric criteria carry evidential weight; multiple evidence channels require deliberate cross-checking and meaningful complementarity; the evidential strength realised in practice may differ from that intended in the assessment design; and automation scales evidence collection more readily than interpretation. The contribution is a conceptually grounded, deployed design for making engineering responsibility and competence more visible when AI can perform substantial artefact production.