SWE-Together: Evaluating Coding Agents in Interactive User Sessions
2026-06-29 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors created a new test called SWE-Together that mimics real-life coding help, where users and coding agents talk back and forth instead of just giving the agent a task once. They gathered real coding sessions and used them to build a system that can replay these conversations with different coding agents. Their test checks not only if the final code works but also how many times the agent needs corrections from the user. They found that better coding agents make fewer mistakes and need fewer fixes, which means a smoother collaboration.
coding agentsinteractive codingbenchmarkuser simulatormulti-turn interactionrepositoryfeedbackLLMcode correctnesscollaborative coding
Authors
Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, Shengzhi Li
Abstract
Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.