Coding agents struggle to fix evolving software requirements reliably
FeatureLoop: Benchmarking Coding Agents under Evolving Requirements
Software EngineeringMachine Learning
Summary
Software coding agents are often given rough instructions that get clearer over time as users test and ask for fixes. The authors created FeatureLoop, a way to test how well these agents handle changing and new coding requests while keeping old features working. They tested top coding models and found that even the best one only finished about 72% of tasks after feedback and that very few errors were successfully fixed quickly. Their findings show that current coding agents have trouble reliably updating code as requirements change and can even undo previous work.
What this means in practice
- •For software development teams: Evaluate and improve coding AI tools that assist iterative software development with evolving requirements.
- •For software test engineers: Use FeatureLoop to identify and measure how coding models handle repairs and regression to guide testing strategies.
Authors
Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu
Abstract
Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.