SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
2026-07-03 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors identify that large language models (LLM) often wait idly for tool results during their reasoning process, which wastes time. They propose SPORK, a system that lets the model predict and start tool calls early, overlapping this with ongoing reasoning to save time. SPORK works without extra training or complex setups by having the model predict its own tool calls with high accuracy. Their tests show SPORK speeds up processing by about 18% without reducing accuracy and works across different model sizes and types.
Large Language ModelsThought-Action-Observation LoopSpeculative ExecutionTool CallsChain-of-Thought DecodingProbe ForkingConfidence GateSpeculative DecodingMixture-of-Experts Models
Authors
Huajun Bai, Weiwei Lv, Huichuan Zheng, Youyou Lu, Jiwu Shu
Abstract
LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.