GitHub Copilot works mostly alone with brief human help
Large-Scale Study of GitHub Copilot Trajectories
Software Engineering
Summary
People often use GitHub Copilot to write code, and this study looked at how much humans actually help during the process. It found that in more than half of the cases, Copilot completed work without any human changes. When humans did step in, they usually made only a couple of quick fixes. Most changes happen early on, and later improvements tend to be small and focused. The study also shows that people sometimes guide Copilot multiple times, even across different versions of the same code changes.
What this means in practice
- •For software development teams: Optimize workflows by understanding when and how developers interact with AI agents like Copilot during code creation.
- •For developer tool designers: Design AI coding assistants that better support early-stage interventions and brief human collaboration sequences.
Authors
Izah Sohail, Yerdanat Kurmantayev, May Mahmoud, Sarah Nadi
Abstract
AI coding agents are increasingly integrated into software development workflows. However, most coding agent studies rely on benchmarks similar to SWE-bench, which assume autonomous agents operating in a single trajectory. This assumption does not mirror real-world usage, where developers interactively collaborate with agents. We address this gap by studying human-in-the-loop agent trajectories in the wild, focusing on 7,581 Copilot-authored PRs with publicly accessible session data. We find that 58% of Copilot PRs have no human intervention, and that when developers do intervene, collaboration is brief (median of 2 Human-AI loops). Merged PRs also tend to have fewer initial trajectory iterations than unmerged ones. We also find that the trajectory following the first human intervention exhibits a significant drop of 20.8 iterations and 8.4 min in duration. However, trajectories following later interventions show minimal iteration drops and no significant duration change, signifying that most of the work happens in the initial trajectory. Our qualitative analysis of illustrative PRs explains this pattern: many developer interventions are small, focused requests that Copilot completes in shorter later trajectories. We also illustrate that developers collaborate with Copilot in multiple ways, including delegated work through feedback as well as adding their own commits. However, we find that developers may need to guide Copilot through multiple (sometimes repeated) interventions in the PR, or even across multiple editions of the PR.