DynBranch speeds up decision-making in dynamic AI workflows
DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
Summary
Some AI programs have steps that only become clear as they run, which slows things down because you have to wait for decisions before continuing. The authors present DynBranch, a way to guess and reuse parts of the work before all choices are finalized. This reduces waiting time without changing how the AI or its tools work. Tests show DynBranch can make such AI tasks run 30% faster or more while still giving correct results.
What this means in practice
- •For ai platform engineers: Reduce response time in AI workflows that decide steps during execution by reusing partial results before full decisions complete.
- •For cloud service operators: Improve throughput and user experience on large language model services with dynamic decision paths by cutting processing delays via speculative subgraph runs.
Authors
Junyi Shen, Noppanat Wadlom, Zhengyuan Su, Yao Lu
Abstract
Agentic LLM workflows decide their execution paths at runtime. Downstream computation may be predictable, or may have run before, yet it cannot begin until the model or the user resolves the branch. We call this serialization the branch-resolution barrier. Caching alone does not hide it: the key that identifies a reusable result is not known until then. In this paper, we propose DynBranch, which makes an unresolved branch addressable before it resolves. Its stable coordinate lets candidate subgraphs run during resolution and completed subgraph results be reused across later requests. A two-level controller admits this work when its expected benefit exceeds the load price. DynBranch sits at the model-API boundary and requires no changes to agent harnesses or model execution engines. Across four agentic workloads with Qwen3-32B on 4x H200 GPUs, DynBranch reduces mean latency by up to 32% over each workload's strongest prior system and by 46-66% against a no-reuse floor, while preserving workflow results. The benefit persists across backbone families and on a commodity Qwen3-8B/RTX 4090 deployment.