Summary
When software agents use external tools to complete tasks, their performance depends both on the agent itself and the restrictions of the environment running it. The authors created ToolBottleneckBench, a way to test how different pressures like available tools or interaction limits affect agent success. They found that limiting which tools are visible to the agent causes the biggest performance drop, mostly due to strict limits on how much the agent can respond. The benchmark helps identify when failures happen and highlights effective fixes outside of changing the agent model. The authors share their benchmark openly for others to use.
What this means in practice
- •For software developers: Evaluate how different resource constraints impact the effectiveness of AI tool agents in enterprise software before deployment.
- •For ai system integrators: Identify bottlenecks due to tool visibility and interaction limits to optimize tool-agent configurations in production environments.
Authors
Zichuan Xu, Yuyang Dong, Makoto Onizuka, Chuan Xiao
Abstract
The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark that measures tool-agent performance under controlled pressures and fixed harness budgets. Synthetic enterprise tasks stay fixed while one pressure varies: the registered tool catalog, the visible tool menu, the certified workflow, or the encoding of returned records. An exact grader records the first unmet obligation of each failed episode and the harness limit, if any, that ended it. Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap. Raising the cap sixteenfold removes most of the penalty in the truncation-heavy configurations. Most failures end at a harness limit, and the failure record identifies the assigned pressure no more reliably than a count of tool calls. The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch. We recommend reporting pressure effects with the harness budgets under which they were measured. We release the benchmark at https://github.com/zichuanxu/tool-bottleneck-bench.