Benchmark tests AI models on real user multimodal coding requests
Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
Artificial Intelligence
Summary
It is hard to check if AI coding helpers understand and fix apps the way real users want, especially when users send mixed types of info like pictures or sound. The authors made Co-Design Bench, a big test set with real user requests and many expert checks to see how well AI models do at these tasks. Their tests show current models can partially meet many requirements but rarely complete all needed changes fully, especially for bigger edits involving multiple changes. This helps understand where AI coding helpers still struggle and where there is room to improve.
What this means in practice
- •For software development teams: Evaluate AI coding assistants on real multimodal user requests to improve their ability to make complex application changes.
- •For qa and testing groups: Use a large, expert-checked benchmark to measure the accuracy and completeness of AI-driven software edits without needing gold code or screenshots.
Authors
Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, Dayiheng Liu
Abstract
Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.