LLMs struggle to build fully working automated workflows from end to end
MCPGen: Benchmarking LLMs on Executable MCPWorkflow Development
Software EngineeringArtificial Intelligence
Summary
The paper looks at how well large language models (LLMs) can create computer workflows that not only look correct but actually run properly from start to finish. The authors found that while models often succeed in putting together the workflow structure, they frequently fail when it comes to making sure all the parts work well together during actual execution. Even when small pieces pass individual tests, complete workflows often do not run successfully. This shows there is still a big challenge in getting AI to reliably build complex, working software processes.
What this means in practice
- •For workflow engineers: Assess and improve LLM-based workflow automation tools by focusing on integration and execution challenges, not just task-level correctness.
- •For software developers: Use MCPGen benchmark to evaluate AI-generated code that builds automated toolchains ensuring cross-layer consistency for reliable execution.
Authors
Yingxuan Yang, Jiaqi Liu, Lirui Guan, Jiaye Gao, Weiwen Liu, Weinan Zhang, Ying Wen
Abstract
We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, and runtime wiring. In this setting, correctness depends on cross-layer consistency: a workflow may be structurally plausible, yet still fail because tool implementations, schema bindings, or runtime execution do not align. Existing benchmarks largely evaluate these capabilities in isolation or rely on trajectory-level proxies, leaving open whether generated workflow artifacts execute end-to-end. We introduce \textbf{MCPGen}, an executable benchmark for Model Context Protocol (MCP) workflow development. MCPGen contains 100 self-contained MCP projects across 16 application domains and evaluates three diagnostic tasks: workflow reconstruction, tool creation, and backward-compatible workflow extension. We evaluate 11 representative LLMs in a single-turn foundation-model setting, assessing generated artifacts through static analysis, unit and integration tests, and process-isolated end-to-end execution. Models reach 88.5\% on workflow reconstruction, but no model exceeds 57\% end-to-end execution success. Per-tool unit-test pass rates reach 63.8\%, while project-level integration success does not exceed 45\%, suggesting that integration remains a major bottleneck even when isolated tool tests pass.