AgentOrch narrows workflow accuracy gap with fewer AI calls
AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?
Artificial Intelligence
Summary
When software agents follow instructions step-by-step, they make decisions after each action, which is accurate but slow and costly. Another way is planning the entire workflow in advance, which saves time and computational effort but is usually less accurate. The authors compare these two approaches on many business workflows and find that the gap in accuracy is decreasing with recent improvements. They suggest methods like building workflows incrementally, revising workflows during execution, and training on complete workflows to make planning nearly as accurate as step-by-step interaction.
What this means in practice
- •For enterprise it teams: Reduce costs and speed up automation by using global orchestration that plans complete workflows with fewer AI calls while maintaining accuracy.
- •For workflow automation developers: Improve automated enterprise workflows by incorporating incremental construction, execution feedback, and targeted training for better reliability.
Authors
Zijie Liu, Qi Yang Huang, Jinhao Duan, Yueh Che Lee, Thomas A. Joseph, Zayne Turner, Adam Seligman, Tianlong Chen, Zhaozhuo Xu
Abstract
Enterprise workflow agents typically rely on step-by-step interaction (Step-by-Step): the agent executes an action, observes its result, and invokes a large language model (LLM) again to decide the next step. Global orchestration (Global) instead generates a complete executable workflow in advance and executes it without further LLM calls, substantially reducing inference overhead, but it remains less accurate than Step-by-Step. We study how close Global can come to Step-by-Step and what narrows the remaining gap. We introduce AgentOrch, a benchmark that directly compares global orchestration with step-by-step interaction on 650 enterprise SOPs and 3,220 runtime variants under matched tools, environments, and evaluation. Step-by-Step remains more accurate for every evaluated model, but the gap has shrunk by about 79% from early models to the strongest recent model. We identify three routes to further narrowing this gap. Incremental workflow construction improves Global accuracy by an average of 9.98 percentage points (pp) without runtime execution feedback. Revising complete workflows with execution feedback improves all evaluated models, including 6.13 pp on held-out runtime variants. Targeted training on complete workflows improves accuracy by an average of 12.18 pp across all five trained models. Although a gap remains, global orchestration offers a promising path toward enterprise agents that deliver higher accuracy with fewer LLM calls.