Summary
Running large language models involves more than just raw computing power as these models get bigger and more complex. The authors show that managing where and how parts of the model run, moving state around, and keeping systems reliable are becoming the hardest problems. They combine insights from software tools and real-world deployments to suggest a new way to plan and control model execution across many machines. This approach goes beyond just optimizing speed to also consider network delays, hardware differences, and service goals. They also offer guidance and open questions about making these systems work well with multiple types of models and hardware.
What this means in practice
- •For cloud infrastructure teams: Manage distributed LLM serving fleets by coordinating execution and state movement to meet performance and reliability targets under varying loads.
- •For ai platform engineers: Integrate model-serving engines with control planes to optimize placement and resource use across heterogeneous hardware for large-scale LLM inference.
A position paper. It proposes an approach and reports no results.
Abstract
Large language model (LLM) inference is evolving from an engine-local optimization problem into a distributed control problem involving reusable state, phase placement, heterogeneous accelerators, networking, autoscaling, reliability, and service-level objectives. This paper connects that transition across peer-reviewed systems research, open-source implementations, and documented production studies. It treats vLLM and llm-d as complementary layers: model-serving engines optimize execution through mechanisms such as PagedAttention, continuous batching, kernels, quantization, and parallelism, while an inference control plane can optimize where, when, and under what policy execution occurs across a fleet. The contribution is synthesis rather than a new benchmark; all reported performance and deployment results remain attributed to their original sources. The combined evidence suggests that the scarce resource in modern inference is shifting from raw FLOPs alone toward managed state, placement, network movement, reliability, and decision quality. We propose an Inference Execution Planner that selects feasible execution plans rather than only endpoints, including aggregated versus disaggregated topology, KV source and transfer action, hardware variant, routing/admission policy, and slower scaling decisions. We also provide a source-local benchmark atlas, a bottleneck-migration taxonomy, practical deployment guidance, an evaluation framework based on SLO-goodput, and research questions for agentic, multimodal, heterogeneous, and resilient inference.