Visual references challenge AI agents in webpage development tasks
RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Building webpages usually involves looking at example websites and using instructions together. The authors created a test called RefCraft that makes AI try to build webpages based on written instructions plus screenshots from real webpages. They found even the best modern AI systems struggle to accurately use these visual examples to make the right webpages. This shows current AI tools find it hard to combine pictures and text when making webpages like humans do.
What this means in practice
- •For web developers: Test new AI tools that combine screenshots with instructions to improve webpage design automation.
- •For ai product teams: Develop practical AI assistants that better interpret visual examples for coding and design tasks.
Authors
Yuchi Wang, Rongyao Fang, Weihao Yu, Xilin Wei, Tiancheng Gu, Dingkang Yang, Hang Zhang, Dayiheng Liu, Hongsheng Li, Shuai Bai
Abstract
Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks. We study this crucial capability through webpage development, a practical and readily evaluable setting in which existing benchmarks remain largely idealized: they typically focus on reconstructing visual targets or generating webpages from textual instructions alone, overlooking the more common workflow of developing from visual examples. Specifically, we introduce RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks that combine natural-language instructions with screenshots from one to five reference webpages. Agents must selectively interpret and utilize these references while rendering, visually inspecting, and iteratively refining their outputs. We evaluate the resulting open-ended artifacts using independently verifiable keypoints and complementary programmatic and VLM judges. Across nine frontier (M)LLMs, the strongest system achieves an overall score of only 52.12 out of 100, revealing that visual-reference-conditioned agentic generation remains a substantial challenge for current models in practical webpage-development settings.