Tool calling speed improved by managing live user memory stores
SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management
Computation and Language
Summary
Calling tools through AI agents usually takes time because each call is processed step-by-step. The authors propose SpecMem, a method that keeps track of past verified tool calls for each user and updates this memory continuously. This live memory helps the system guess and speed up future calls without making errors. SpecMem also carefully checks before making irreversible tool calls, avoiding mistakes. Tests showed this approach made tool calls faster and more accurate compared to previous methods.
What this means in practice
- •For ai system developers: Speed up AI tool integrations by maintaining and updating verified per-user tool-call memories to reduce latency.
- •For customer support platforms: Improve response time for recurrent user queries by leveraging live memory stores of past verified requests.
Authors
Zihe Song, Wenting Zhao, Ye Liu, Yuanjie Zhu, Ke Xu, Shanghao Li, Philip S. Yu
Abstract
Tool-calling agents are mostly optimized for accuracy; in deployment, however, their growing cost is latency: every tool call is decoded autoregressively. Speculative decoding accelerates this without changing the output, and retrieved past calls make strong drafts under real use, where each user's behavior tends to repeat. Yet existing systems keep no cross-session memory, or build the draft store once and never update it. We propose SpecMem, a tool-call memory kept alive and managed throughout deployment: verified calls are written back online into bounded per-user stores with LRU eviction, and a side-effect gate defers irreversible calls until verification confirms. On multi-session, multi-user query streams served by three open-weight models, a frozen store steadily loses acceptance while SpecMem holds it: a 27% gap in mean accepted tokens (+11% to +34% across benchmarks, tracking how much novel work arrives after the store is built) and up to 1.74x wall-clock speedup. Ablations attribute the gain almost entirely to keeping the store live; local per-user scoping adds no acceptance but closes a measured cross-user leakage channel, and gating on declared side effects, rather than on retrieval confidence, is what eliminates irreversible mistakes.