EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants
Artificial IntelligenceSoftware Engineering
Summary
The authors created EvoGenUI-Bench, a test to see how well language models can keep web interfaces working correctly over several steps of changes. They tested 150 tasks where the interface must update and still function properly over five turns. By running the interfaces in a browser and checking various outputs, they found that even the best models often fail to keep all parts working together through multiple changes, especially when the interface relies on external tools or states. Their study highlights that evaluating such AI-generated interfaces needs to focus on how well behavior and state stay consistent as the interface evolves, not just on single outputs.
Authors
Yue Peng, Lanke Xia, Zihan Wang, Jiahao Ye, Ke Ning, Hongyi Wen
Abstract
Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve. We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state. We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs. Beyond turn-level and episode-level success, we measure cross-turn retention with Adjacent Pass Retention. Across eight models, even the strongest achieves 74.9% Turn Pass while completing only 37.3% of five-turn episodes; APR further falls to 52.4% on tool-grounded tasks. Diagnostic analysis shows that presentation failures center on information architecture, interaction failures on derived-state propagation and affordance binding, and tool-grounded failures additionally involve external-state grounding and requirement decomposition. These results reframe generative UI evaluation from judging isolated outputs to testing whether interface behavior, derived state, external state, and assistant claims remain synchronized as the artifact evolves.