Coding agents often miss full product requirements in enterprise projects
When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories
Software EngineeringArtificial Intelligence
Summary
Coding agents are tools that write software automatically based on instructions called product requirement documents (PRDs). The authors found that these agents often create parts of a project but fail to complete or properly connect all pieces of large enterprise software tasks. They made a new test called PACE to measure how well agents deliver on entire PRDs and identify where they go wrong. Using this test, they saw the best agents only fully completed about 14% of tasks, with many missing integration steps.
What this means in practice
- •For enterprise software developers: Evaluate and improve coding agents’ ability to fully meet product requirements in complex enterprise projects.
- •For software quality assurance teams: Use the PACE benchmark for evidence-based assessment of integration gaps and partial implementations from AI coding agents.
Authors
Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv
Abstract
Coding agents increasingly undertake development tasks in mature enterprise repositories, yet existing benchmarks offer limited insight into whether they fulfill complete Product Requirement Documents (PRDs) and where delivery breaks down. Issue resolution and development from scratch leave incremental PRD fulfillment underexplored, while isolated tests provide incomplete evidence of business and integration requirements. We introduce PACE, a benchmark of 43 enterprise tasks with 750 expert-audited Code Points. It combines requirement reconstruction, criterion construction, and evidence-based evaluation to measure PRD fulfillment and connect delivery gaps to agent behavior. Across 11 model--scaffold configurations, most scores fall between 52.1 and 78.1; GPT-5.6-Sol achieves the highest complete-delivery rate of 14.0\%. Incomplete implementation and missing integration are the principal bottlenecks: producing individual components does not ensure a complete delivery. Evaluation achieves 95.2\% agreement with expert consensus, with lower case-level variation across judging records than repeated executions. Beyond model comparison, the benchmark provides actionable feedback for harness refinement: diagnosis guides targeted improvements, and successive harness releases raise mean performance from 59.4 to 63.0 using the same model and task suite. Code is available at https://github.com/TianqiLiu1/PACE-Bench.