Developers interact differently with ai coding tools depending on understanding and testing
Beyond the Prompt: Linking What Developers Ask, Do, and Understand with Coding Agents
Software Engineering
Summary
Developers using AI tools to write code don’t all work the same way. The authors studied how developers ask questions, what they do while the AI works, and how well they understand the results. They found that some developers rely more on the AI to check their code, while others test more themselves. Understanding these differences can help improve how people and AI tools work together in programming.
What this means in practice
- •For software development teams: Improve collaboration between developers and AI code tools by recognizing different usage styles and tailoring support accordingly.
- •For software engineering educators: Use the workflow to assess how students interact with coding assistants, capturing prompt use, actions, and understanding to guide teaching.
Tested on one dataset.
Authors
Yunhan Qiao, Summit Haque, Christopher Hundhausen
Abstract
Coding agents can now change code for developers, who describe goals, supply context, and respond to the agent's work. Yet prompts, screen activity, and task success each tell only part of this story. We present Say, Do, Understand, an end-to-end workflow for analyzing what developers write to an agent, what they do while it works, and what they can explain afterwards. The workflow has five stages (Capture, Prepare, Analyze, Integrate, and Interpret) and three instruments: a prompt codebook, a scheme for coding screen-recorded activities and events, and separate rubrics for explaining the process and the solution. We applied it in an observational study of ten experienced developers who used GitHub Copilot on an unfamiliar codebase. Crossing task performance with understanding produced four personas. The two measures agreed for eight developers but split for two: one passed most tests but could not explain the solution, and another passed few tests but explained it well. In this sample, the personas that most often asked the agent to check its work spent the least time testing on their own. These patterns are descriptive and do not generalize beyond the sample. We recommend the workflow to computing educators, industry practitioners, and researchers to adapt and evaluate human--AI communication in software engineering.