To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks
2026-06-29 • Human-Computer Interaction
Human-Computer InteractionArtificial IntelligenceSoftware Engineering
AI summaryⓘ
The authors studied how students use AI tools that suggest code while they program, like Github Copilot. They created a tool called Clover to track how students interact with these suggestions and included simple tests to see if students were thinking carefully. They found that quickly accepting suggestions was linked to less careful thinking, while spending more time on the suggestions was linked to more careful thinking. The authors suggest these insights could help promote better understanding when using AI to help write code.
AI code completionGithub Copilotprogramming educationuser interaction loggingattention checksreflective engagementbehavioral metricstab acceptdwell timeprogramming tasks
Authors
Jessica Hutchison, Ian Tyler Applebaum, Kenneth Angelikas, Kush Rakesh Patel, Phuoc Nguyen, Antonio Lazaro, Nicholas Rucinski, Rahad Arman Nabid, Stephen MacNeil
Abstract
AI code completion tools, such as Github Copilot, provide students with code suggestions to help them write programs. However, recent qualitative studies suggest that students fail to critically evaluate these suggestions. We present Clover, a code completion tool that logs students' interactions with code suggestions and additionally offers attention checks to probe reflective engagement during programming tasks. We also develop a taxonomy of behavioral interaction metrics for AI-assisted programming, informed by literature. We analyzed relationships between interaction patterns, engagement with attention checks, and task performance. We observed that higher rates of tab accept were associated with lower attention check performance, while increased dwell time was associated with higher attention check performance. We conclude by discussing how programming process data and attention checks might support reflective engagement in AI-assisted programming.