Tool use failures in AI depend on state and scheduling limits
What Structurally Breaks an Agent? Auditing Difficulty in Stateful Tool-Use Workflows
Machine Learning
Summary
It’s hard to know why AI agents fail at tasks that need many steps and tools. The authors created a testing tool that changes factors like how many choices an AI sees or how it keeps track of past info to find out what breaks an agent’s workflow. They ran over 700,000 tests and showed that some problems come from limits in remembering and scheduling steps, while others cannot be fixed even with better tracking. Their tool helps pinpoint these weak spots in AI workflows.
What this means in practice
- •For ai system engineers: Diagnose and isolate failure modes in complex AI workflows involving stateful tool usage by testing different memory and scheduling parameters.
- •For software developers building ai assistants: Improve design of AI assistant workflows by understanding which state tracking and scheduling limitations cause breakdowns in multi-step tool use.
Authors
Shereen Anand, Ju Wang, Yali Du
Abstract
An aggregate task score does not distinguish failures of scheduling, state retention, and candidate selection. We develop a small executable instrument that varies dependency depth, candidate count, active state width, and access to current-state snapshots while holding the number of required operations fixed. Two dependency-graph families show that increasing depth at fixed work changes both scheduling freedom and the number of reachable completion states, so these controls are coupled, and we quantify the coupling. The primary study contains 49,152 generated tasks and 393,216 reference-controller rollouts. A separately frozen follow-on design adds 295,936 rollouts covering retention strategy, snapshot cadence, hidden state resampling, recovery, query locality, candidate indexing, and relabeling. Residue caches and precomputed plans solve tasks that defeat limited raw-value caches, and indexed selection removes the serial-scan branching deficit. Hidden resampling imposes an exact success ceiling that additional state tracking cannot overcome. The contribution is an executable diagnostic instrument, analytic calibration, and an audit of its assumptions. A small local CPU study with a 0.5B model completed no workflows, and a 16-episode Copilot-routed fractional factorial is descriptive only. Neither supports general capability claims.