ChatGPT usage diversity linked to better security assignment scores
Exploring the Role of Security Experience and ChatGPT Usage Strategies on Secure Software Engineering Education
Software Engineering
Summary
Using ChatGPT is becoming common in software education, but how it helps with security tasks is not fully known. The authors studied 26 cybersecurity students using ChatGPT to fix code problems. They found that it wasn’t just whether students used ChatGPT that mattered, but how many different ways they used it. Students who used a wider variety of ChatGPT strategies tended to get better grades, regardless of their prior experience. This suggests teaching students to interact with AI tools in different ways might improve learning about secure coding.
What this means in practice
- •For software engineering trainers: Design training sessions that encourage multiple ways of using AI tools to improve secure coding assignment outcomes.
- •For cybersecurity training providers: Develop curricula that integrate diverse AI usage strategies to enhance student performance in vulnerability assessment tasks.
Authors
Alessio Ferrari, Minh An Nguyen, Kushal Ramkumar, Liliana Pasquale
Abstract
The rapid adoption of Large Language Models (LLMs) is reshaping software engineering education, but their role in secure software engineering education remains underexplored. We report an exploratory empirical study of how 26 graduate students in a part-time MSc Cybersecurity programme used ChatGPT during a vulnerability-fixing assignment. To characterise ChatGPT use, we analysed students' ChatGPT interaction logs using a structured double-coding procedure and examined whether usage patterns and prior cybersecurity expertise were associated with assignment performance. The results show that students with varying levels of cybersecurity expertise used broadly similar ChatGPT strategies. Individual usage patterns showed descriptive differences by grade, but none remained statistically significant after correcting for multiple comparisons. In contrast, diversity of ChatGPT usage, i.e., the number of distinct usage patterns adopted, was positively associated with performance, even after controlling for cybersecurity expertise. These exploratory findings suggest that the way students engage with ChatGPT may be more informative than whether they use it, and motivate future controlled studies to guide students toward effective LLM use in secure software engineering education.