Developers struggle to spot security flaws in AI generated code
(Don't) Trust, but (Don't) Verify: Developers' Attention to Security in AI-Generated Code
Cryptography and SecuritySoftware Engineering
Summary
AI tools can help programmers write code but sometimes produce unsafe code. The authors studied how well developers notice security problems in AI-generated code and what influences their trust in it. They found many developers had difficulty identifying vulnerable code and relied on unclear cues. This study shows the challenge of safely using AI helpers in coding and points to where developer attention is focused.
What this means in practice
- •For software developers: Improve review of AI-generated code by focusing on cues that signal security risks to reduce vulnerabilities in final code.
- •For security engineers: Develop training or tools that help identify weak points in AI coding suggestions to enhance secure software development.
Authors
Hamza Khalid, Ronald E. Thompson, Alejandra Sabater, Perucy Mussiba, Kelsey R. Fulton, Daniel Votipka
Abstract
AI coding assistants are rapidly transforming software development, but are known to produce insecure code. Prior work has measured whether AI-assisted developers produce secure code, but less is known about how they evaluate AI-generated code: whether they can identify vulnerabilities, what cues they use, and how trust shapes their decisions. This evaluation step is foundational to secure development with AI, whether using auto-complete, chat tools, or AI agents. As a first step, we conducted a remote observational study with 100 participants isolating this evaluation stage. Participants were tasked with producing secure and functional code for four C linked-list tasks. For each, participants were able to cycle through five AI-generated suggestions varying in security and functionality, select one, and edit their choice into a final submission. Participants also completed a post-study survey about their decision-making and perception of AI-generated code's security and 23 completed a more in-depth interview.