Coding agents clarify incomplete client requests for software delivery
SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?
Software EngineeringArtificial IntelligenceComputation and Language
Summary
Building software from scratch often starts with clients giving incomplete instructions, making it hard to know what exactly to build. The authors created a new test called SWE-Clar using real freelance projects to see how well AI coding agents can ask the right questions and then write the software accordingly. Their tests show that even the best AI models only successfully complete about two-thirds of the tasks correctly. This highlights that fully independent software building by AI from unclear requests remains very challenging.
What this means in practice
- •For freelance software developers: Test AI tools that clarify client requirements and improve project success on freelance platforms.
- •For software product managers: Evaluate how AI can independently gather requirements and deliver code for new software projects.
Authors
Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen
Abstract
Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two benchmark design choices: clarification precedes code implementation, and its effectiveness is assessed through explicit tests of requirement understanding. Guided by these choices, we introduce SWE-Clar, a benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each with at least US\$4,000 in total payments. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client-freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even GPT-6 Astra, which has the highest observed implementation score, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. The dataset and evaluation code are available at https://github.com/upwork/SWE-Clar.