Frontend coding agents can break websites despite completing tasks
ProductWebBench: Evaluating Website Continuity Beyond Task Completion
Software EngineeringArtificial Intelligence
Summary
Building new features on a website isn’t just about adding the right parts; the rest of the site has to keep working smoothly as well. The authors introduce ProductWebBench, a test system that checks if code changes not only meet the task but also keep the website’s other functions working without errors. They found many code changes pass the main task but still cause problems like broken pages or errors unseen until simulated users interact. This shows evaluating code by task completion alone misses important real-world issues.
What this means in practice
- •For frontend developers: Validate that code changes keep the entire website working, not just complete requested features, to reduce hidden bugs before release.
- •For quality assurance teams: Detect regressions and user interaction failures caused by frontend code changes using a benchmark that simulates realistic website usage.
Authors
Jiajun Wu, Jian Yang, Yaxin Du, Shukai Liu, Linzheng Chai, Tuney Zheng, Wei Zhang, Haitao Yang, Yan Bai, Yihang Lou, Xianglong Liu, Bryan Dai
Abstract
Frontend coding agents are usually graded on one question: did the requested target get built. Real engineering changes carry a second obligation, because the rest of the site has to keep working. We call that second obligation website continuity, and introduce ProductWebBench, a benchmark that scores both on living frontend repositories: 320 Change tasks over existing code and 80 long-horizon Build tasks. Every task freezes a repository snapshot, staged requirements, browser evidence and deterministic checks before evaluation, and enters only once a reference passes every check and plausible-wrong ones fail. A simulated user reveals the requirements one stage at a time, and every check cleared earlier stays armed for the rest of the run. Across 8,737 runs from 13 models the two come apart: of the Change runs passing every requested-work check, 28.2% still fail a continuity check -- a blank route, horizontal overflow, console errors -- and 10.8% push a browser state they had already passed back to failing. Because the same checks also run on the untouched repository, each regression carries exact provenance, not a post-hoc label: on the 586 runs where both readings exist, 18.3% are caught breaking a verified state. Interaction is a further blind spot, failing in 29.5% of runs whose static DOM already passes. ProductWebBench therefore scores a patch against the living system it lands in, not the target alone.