Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
2026-06-01 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingHardware ArchitectureMachine Learning
AI summaryⓘ
The authors studied how large language model (LLM) agents handle tasks that involve many steps, where the total work is unknown at the start. They found that treating each conversation as a single unit, instead of individual turns, creates a simpler, two-part process: an initial heavy computation at the start, followed by a longer memory-focused phase. Their method, called ConServe, uses this idea to manage resources more efficiently without needing predictions about each step's cost. Compared to older methods, ConServe cuts initial delay by about half and improves energy use, especially when using different types of GPUs for the two phases.
LLM agentsmulti-turn inferencescheduling unitprefilldecodeKV cachelatencyenergy efficiencyGPU tiersConServe
Authors
Jianru Ding, Ryien Hosseini, Pouya Mahdi Gholami, Mingyuan Xiang, Henry Hoffmann
Abstract
LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives. Existing multi-turn systems keep the turn as the scheduling unit and decide, turn by turn, whether to disaggregate prefill from decode. That decision rests on the turn's decode length, tool behavior, and KV growth, quantities that are not observable when the scheduler must act, forcing the system to predict them. We show this dependence on prediction is imposed by the scheduling unit, not the workload. Raising the scheduling unit from the turn to the conversation converts turn-level irregularity into a stable, two-phase structure: 1) a compute-bound turn-1 prefill followed by 2) a long, memory-bound tail. Thus, with the conversation as the scheduling unit, placement reduces to reading the first-turn input length and per-decoder KV occupancy, both directly observable. We instantiate this principle in ConServe, which routes the first-turn prefill to a high-throughput prefiller, transfers the KV cache exactly once, and pins the conversation to a single decoder for its entire tail, with no learned model of decode-side cost. Against a per-turn prediction baseline, ConServe reduces p95 time-to-first-effective-token (the latency of a conversation's first user-visible output) by 51.08% and improves energy efficiency by 7.51% while preserving last-turn TBT and SLOs; mapping the two phases onto heterogeneous GPU tiers adds a further 22.75% in energy efficiency.