OpWeave lowers large language model serving cost on mixed GPUs
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
Running large language models (LLMs) on computers with different types of GPUs can be expensive and slow. The authors created OpWeave, a system that breaks down the model’s work into smaller parts and smartly assigns these parts to different GPUs. This careful planning helps use the hardware more efficiently and save money while keeping response times fast. Their tests show that OpWeave reduces costs significantly compared to previous methods.
What this means in practice
- •For cloud infrastructure teams: Cut operational costs of running LLM inference pipelines on clusters with diverse GPU types while meeting latency goals.
- •For data center operators: Optimize load distribution of large ML workloads across heterogeneous GPU clusters to improve resource use and reduce serving expenses.
Authors
Zikun Li, Yixuan Mei, Shiqi Pan, Zixuan Chen, Xiaowen Zhang, Mengdi Wu, Shuhuai Lin, Yutong Yang, Zhihao Zhang, Xupeng Miao, Rashmi Vinayak, Zhihao Jia
Abstract
LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to $1.78\times$ on homogeneous and $1.89\times$ on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.