Automatic tool updates improve agent system success and cut costs
AT-AT: Automatic Tool Optimization from Agent Traces
Machine Learning
Summary
Agents that use large language models often rely on tools to complete tasks, but improving the agents alone isn't enough. The authors present AT-AT, a system that improves these shared tools by learning from past agent interactions without changing the agents themselves. By analyzing how agents use tools and their success rates, AT-AT tweaks the tool descriptions and implementations to help agents work better and cheaper. Tests showed AT-AT raised task success rates and reduced errors across different domains and even in real-world enterprise servers.
What this means in practice
- •For enterprise software teams: Improve existing shared toolsets used by automated agents to reduce errors and operational costs without changing the agents themselves.
- •For api maintainers: Automatically refine API descriptions and implementations using historical usage data to increase reliability and task success.
Authors
Maya Ravichandran, Pragya Tripathi, Dharamendra Kumar, Adith Swaminathan, Nathan Kallus
Abstract
Efforts to improve LLM-based agents typically focus on modifying the agent itself, yet success also depends on the environment in which an agent acts. Tools mediate this interaction: a tool's description shapes how an agent calls it, and its implementation determines the results and errors the agent receives afterwards. Many tools, such as APIs and MCP servers, are shared by many agents and maintained by providers who cannot change those agents. We introduce AT-AT (Automatic Tool Optimization from Agent Traces), an agentic tool optimizer that improves a shared toolset offline from historical agent executions while holding the agents fixed. A lightweight coding agent reads agent trajectories and their evaluation results, and selectively edits tool descriptions and, when permitted, implementations. On $τ^2$-bench-verified, AT-AT achieves the highest task completion rates in the Airline and Telecom domains at lower cost than strong prompt- and tool-optimizer baselines, and its Telecom gain is statistically significant under paired permutation tests. Optimizing the tools within a coding agent harness also improves held-out performance on TerminalBench-2 and TBLite. When applied to three production MCP servers at a large enterprise, AT-AT lowered cost or tool error rates on every server and increased held-out task passes on one. By turning accumulated execution traces into updates to shared tool environments, AT-AT provides a path toward continual improvement of agent systems through environment-side learning.