Coding agents perform differently when mimicking real user task flows
Coding-Agent Benchmarks Should Match Their Users' Task Flows
Machine LearningArtificial IntelligenceComputation and LanguageSoftware Engineering
Summary
Many computer programs help software engineers by writing or analyzing code, but testing these helpers often uses tasks that don’t match how engineers actually work. The authors studied thousands of real coding sessions and found that engineers switch between many types of tasks, like asking questions, planning, or reviewing code, within a session. They propose a way to make testing tasks better match these real workflows, which affects how well the helpers perform and how much they cost to use. This helps make evaluations of coding tools more realistic and useful.
What this means in practice
- •For software engineering teams: Use task flow aligned benchmarks to better assess coding assistants during real-world development workflows.
- •For ai tool developers: Design coding agent evaluation protocols that reflect realistic multi-step, mixed-type user interactions for improved agent tuning.
Authors
Igor Slinko, Yaroslav Golubev, Sergey Titov
Abstract
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.