Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle
2026-07-03 • Cryptography and Security
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
AI summaryⓘ
The authors explain that AI systems are now being used to write, secure, and test software all by themselves. However, they warn that having the same AI do all these jobs can cause shared blind spots, making it hard to trust the results because there is no independent check. They argue that humans should always stay involved in these processes to catch mistakes, keep accountability, and stop attackers from exploiting predictable AI behavior. The paper suggests a thoughtful balance between human oversight and machine automation is necessary for safer software development.
Artificial IntelligenceGenerative ModelsSoftware TestingVerificationHuman-in-the-loopAdversarial Machine LearningAutomationCode GenerationAccountabilitySoftware Security
Authors
Mohamed Chahine Ghanem
Abstract
Artificial intelligence has spread across the whole of the security lifecycle. The same family of models now writes application code, hardens it, and probes it for weaknesses, so that a single generative substrate increasingly performs all three roles at once. Enthusiasm for this convergence tends to treat full autonomy as the natural end point of partial assistance. This article argues that it is not. When the system that builds an artifact is drawn from the same distribution as the systems that defend and test it, the three roles inherit a common set of blind spots, and the independence that makes verification meaningful is quietly lost. Removing the human does more than raise the automation level: it collapses the external oracle against which machine output is judged, outruns the point at which a person could intervene, hands adversaries a predictable and poisonable target, and dissolves the locus of accountability when something fails. Drawing on evidence from autonomous code generation, adversarial machine learning, software fault tolerance, and the first all-machine hacking tournaments, we argue that the human belongs in the loop not as a temporary scaffold but as a permanent structural requirement, and set out what a defensible division of labour between people and machines should preserve.