Interactive benchmark measures quality of AI-generated web apps
IWC-Bench: Evaluating Web Application Generation from a Software Testing Perspective
Software EngineeringArtificial Intelligence
Summary
Testing how well AI can build web applications is hard because simple checks may miss problems or unused features. The authors created IWC-Bench, a tool that tests AI-generated web apps by simulating real user actions to explore all features and then scores the app based on look, ease of use, and how well it meets requirements. This approach avoids missing hidden bugs or features and better matches what humans think about app quality. Their tests show IWC-Bench agrees well with human judgments and works consistently across different AI models.
What this means in practice
- •For web developers: Evaluate AI-generated web applications by automatically testing their functionality and user experience comprehensively without missing hidden features.
- •For software testing teams: Use guided interaction traces to improve automated testing coverage and defect detection in AI-produced software products.
Authors
Chenxu Liu, Zilu Zou, Peizhong Gao, Jiawen Tao, Zhexin Zhang, Guang Chen, Haowei Lin, Ying Zhou, Tianyi Bai, Dolly Deng, Suncong Zheng, Maxm Pan
Abstract
Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose IWC-Bench, an interactive benchmark for evaluating web application generation from a software testing perspective. IWC-Bench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, IWC-Bench collects runtime evidence without constraining exploration to predefined acceptance criteria. IWC-Bench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 16 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, IWC-Bench achieves 85.3\% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.