Coding agents tested with dynamic multi-turn user simulations
SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
Software EngineeringArtificial Intelligence
Summary
Typical testing for coding assistants uses single, clear questions, but real users ask many questions one after another, adding new details each time. The authors created SWE-DIVE, a system that acts like a user having a back-and-forth chat with a coding assistant to better test how well the assistant works in real-life conversations. This system also checks if the assistant solves the task correctly by following the evolving requirements. This approach helps to evaluate coding agents in a more realistic and interactive way.
What this means in practice
- •For software development teams: Test interactive coding assistants in realistic scenarios where requirements change through multiple interactions.
- •For application developers: Improve multi-turn chatbots by validating their ability to handle evolving user requests dynamically.
Authors
Lorenz Brehme, Benjamin Steenhoek, Ruth Breu
Abstract
Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a single prompt; it typically involves a multi-turn interaction in which the user iteratively adds new tasks and requirements that the agent must fulfill. To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment. To achieve this, we introduce a user simulator that can simulate different behavioral patterns and communicate with the coding agent. We also introduce a conversation validator that evaluates the user simulator's outputs to determine whether they fulfill the initial requirements and guide the agent toward successfully solving the task. The entire framework is embedded in a Harbor environment, which orchestrates the coding agent, user simulator, and conversation validator, and executes the test environment to determine whether the agent successfully solves the task. Our results and source code are available in our GitHub repository (https://github.com/lorenzbrehme/swe-dive_usersim).