Agent tool calls let servers track progress to save memory
Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceOperating Systems
Summary
When computer programs called agents ask tools for help, they often wait a long time while holding on to important memory. The authors show it's very hard to guess how long these tools will take before they start running. Instead, the tools can report their actual progress while running, and this lets the servers make better decisions about memory use. The authors tested this idea and found it made memory management much more efficient and reduced waiting times significantly.
What this means in practice
- •For ai platform engineers: Improve response speed and reduce memory usage when AI agents call external tools by integrating tool-reported progress signals.
- •For cloud service operators: Manage GPU memory allocation more precisely during agent tool interactions to lower latency and resource waste in cloud AI services.
Authors
Yipeng Liu, Yingqiang Zhang, Feifei Li, Huanchen Zhang
Abstract
An agentic request spends substantial wall-clock time waiting for tools, and its KV cache holds GPU memory the whole time. Serving systems decide whether that cache stays, leaves, or comes back by guessing how long the tool will run, from the tool's name, its history, a duration declared before the call, or the engine's own occupancy. We show that no estimate fixed before a call starts can know its duration, and such estimates may not even rank the calls. Meanwhile, the running tool already holds the answer, but the agent stack together with the tool silences it. We propose that tool calls report their progress explicitly while they run, and we measure what that takes. A census of four public agent corpora finds a readable signal in most tool time once it is revealed, in two strengths: a fraction of the work remaining, or an accurate signal that the end is near. A harness recovers it without changing what the agent sees, at no measurable cost to the agent's benchmark score. At the points where a KV cache decision is made, the reported progress is between several times and an order of magnitude more accurate than the best published predictors, and it stays accurate when the environment changes. Plugged into a production engine through a few small hints, it cuts the p90 time to first token (TTFT) after a tool call by 20.7% (HBM only) and 20.8% (HBM + DRAM) against LRU, close to an oracle. A serving system should not guess what its tools can tell it.