Benchmark measures how ai agents organize complex multi-step tasks
WorkflowArena: Benchmarking Executable Workflow Orchestration in LLM-Based Multi-Agent Systems
Computation and LanguageSoftware Engineering
Summary
Many AI systems use multiple agents working together to solve complicated problems. The authors created a new benchmark called WorkflowArena to test how well these AI agents can organize and manage their teamwork on long, complex tasks across several professional fields. They measure not just the final results but how the work is divided and delegated among agents. Experiments showed different AI models have unique ways to split tasks, and better organization can improve overall performance.
What this means in practice
- •For software development teams: Test and improve AI helpers that coordinate complex coding and debugging tasks across multiple agents in software projects.
- •For legal document analysts: Evaluate AI systems that autonomously organize workflows to break down and analyze intricate legal tasks with multiple interdependent parts.
Authors
Zheng Jiang, Fang Niu, Bo Peng, Houde Qian, Wenyi Li, Deyao Hong, Mingju Gao, Yizhe Chi, Youjie Zheng, Kaisen Yang, Calvin Xiao, Lifeng Sun, Huan-ang Gao
Abstract
Large language model agents are increasingly deployed as multi-agent systems rather than as a standalone model operating within a single context. Recent agent harnesses allow the model to instantiate worker agents and to author the executable workflow that coordinates them, shifting the design of system structure from developers to the model at run time. The quality of this organization therefore constitutes a distinct capability, yet existing benchmarks either measure end-to-end task completion or evaluate coordination under structures specified in advance. We formalize this capability as workflow orchestration and introduce WorkflowArena, a benchmark of 145 long-horizon tasks spanning 8 professional domains, from software engineering and formal mathematics to legal analysis and 3D production. Each task specifies an initial workspace and an integrated deliverable composed of interdependent components, which is assessed by a hidden verifier. Agents determine autonomously whether and how to delegate, and all delegation is recorded, enabling joint evaluation of deliverable quality and the organization of the underlying work. Experiments with nine models show that no single configuration performs best across all domains and that systems exhibit distinct delegation strategies: at high effort, Claude models in Claude Code invoke workflows less frequently but with larger worker groups, whereas GPT models in Codex delegate more frequently in smaller units. When the executing model is held fixed on a challenging subset of tasks, workflows authored by Claude Opus 5.5 substantially improve weaker executors, indicating that workflow organization is itself a measurable determinant of task performance.