LLMs struggle to clarify vague software requirements through interaction
Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development
Software EngineeringArtificial Intelligence
Summary
When people ask AI to write software, they often start with unclear ideas. This can lead to mistakes if the AI guesses wrong because the instructions aren’t clear enough. The authors created a test with many software tasks to see how well different AI models ask questions to understand what users want before coding. They found that even if an AI asks good questions, it doesn’t always produce better requirement documents, and performance changes depending on the user's style. This work helps understand how AI can improve back-and-forth communication during software creation.
What this means in practice
- •For software development teams: Improve tools that guide AI to ask better questions and clarify vague requests before coding large software projects.
- •For quality assurance engineers: Use benchmarks to evaluate how well AI tools understand requirements and reduce costly errors in complex software tasks.
Authors
Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He
Abstract
As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause costly rework. However, existing benchmarks typically evaluate implementation from complete specifications or underspecified requests, overlooking pre-coding requirement clarification. Prior clarification studies mainly focus on function-level tasks. It therefore remains unclear how well LLMs can transform vague intent into clear requirements through interaction and how this capability affects subsequent code generation. We introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. It contains 341 tasks spanning game, miniapp, and web development, each with a vague user intent, a human-validated requirement document, and code evaluation artifacts. We employ LLM-based user simulators with diverse personas derived from real-world conversations. We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality. Results show that stronger clarification effectiveness does not necessarily translate into higher document quality, and performance varies substantially across user personas. We also find that coding-agent harnesses do not further improve requirement document quality compared with their base LLMs. These findings offer insights for building LLM-based end-to-end software development systems.