DolphinRouting improves disaggregated large language model serving efficiency
DolphinRouting: Cluster-Level Binary Routing for Disaggregated LLM Inference Serving
Machine Learning
Summary
Serving large language models often involves splitting work into two parts: preparing the input (prefill) and generating the output (decode). Choosing where to handle these parts—locally or remotely—is tricky because it depends on many factors like cache use and request size. The authors designed DolphinRouting, a system that learns the best place to do prefill work for different groups of requests, improving response speed and efficiency. They tested it on a real cluster and showed it reduces latency compared to previous methods.
What this means in practice
- •For cloud infrastructure engineers: Optimize routing of inference requests in disaggregated LLM serving to reduce tail latency and improve throughput.
- •For machine learning platform operators: Improve resource allocation decisions for prefill versus decode tasks based on learned clusters of request characteristics.
Authors
Yishuo Yuan, Shaoyu Yang, Chunrong Fang
Abstract
Prefill-decode (PD) disaggregation isolates the two phases of large language model serving, but request-level path selection must account for prefill work, cache reuse, and decode demand. We present DolphinRouting, which learns this choice from end-to-end SLO feedback under a given P/D resource allocation. Requests are clustered by raw prompt length, cache-adjusted input lengths on both worker pools, and output-length information. Offline policy search learns one binary action per cluster. Online feature extraction, cluster assignment, and table lookup select local or remote prefill. We evaluate DolphinRouting on a real PD-disaggregated cluster using three public benchmarks, a mixed workload, and real data. Across the mixed workload and real data, DolphinRouting with $k$-means reduces P90 TTFT by 5.7--7.9\% and P90 TPOT by 2.6--10.1\% relative to Dynamo.