Summary
Enterprises want AI agents to automatically use many different software tools to complete long tasks, but current tests only check simple or few-step tool use. The authors created SLATE, a new benchmark that uses about 1,000 e-commerce APIs and realistic simulated responses to test AI models on picking the right APIs across multiple steps for complex tasks. SLATE also measures success automatically by checking if the entire task outcome is achieved, allowing different valid ways to complete the task. They found even the best AI models struggle, completing only 5% of tasks, showing that selecting tools for complex, long jobs is still very hard for these systems.
What this means in practice
- •For enterprise software teams: Evaluate and improve AI agents that automate complex workflows across many internal APIs with longer task sequences in realistic conditions.
- •For enterprise ai platform builders: Test and compare architectures for AI-driven API tool use in internal enterprise ecosystems to build more reliable automation solutions.
Authors
Rongzhe Wei, Ge Shi, Min Cheng, Sarthak Ghosh, Vaibhav Gorde, Leman Akoglu, Pan Li, Na Zhang
Abstract
Enterprises increasingly want LLM agents to automate standard operating procedures over their software stacks -- mapping each step of a procedure to the right call among thousands of internal APIs, without hand-authored step-to-API mappings. The core difficulty is API selection at scale: choosing and composing the correct tools over a long horizon, judged only by whether the overall task succeeds. Current tool-use benchmarks do not test this regime -- they expose only a handful of tools, evaluate single-step invocation, or grade multi-step behavior with subjective LLM-as-a-judge scores that penalize valid alternative trajectories. We introduce SLATE (Synthetic Large-scale API Toolkit for E-commerce), a large-scale benchmark that tests large-toolset, long-horizon API selection in a realistic enterprise domain. SLATE couples (i) queries decomposed into hierarchical, tool-grounded procedures over a ~1,000-tool e-commerce library, (ii) a deterministic, context-aware simulator that returns recorded outputs for matching calls and default responses otherwise, and (iii) an end-to-end task-success metric -- a task succeeds when it reaches every reference outcome -- that credits diverse-but-valid trajectories while remaining fully automatic and reproducible. Experiments with six open decision models and the language models they are built on, from 3B to 12B parameters, show that even the best model completes only 5.0% of the tasks, establishing SLATE as a challenging testbed for reliable enterprise tool use. We release SLATE, its simulator, and the evaluation harness at https://geshijoker.github.io/slate/