NebulaSD improves large language model speed with flexible batch sharing
NebulaSD: Many-for-Many Speculative Decoding
Networking and Internet Architecture
Summary
Large language models take time to generate text because they check each word one by one. To speed this up, a small helper model guesses several words ahead, while the main model quickly checks if those guesses are right. The authors found that fixing the helper and main model to work together can waste resources, especially when many requests happen at once. Their system, NebulaSD, lets helper and main models work independently and share the workload more flexibly. This leads to much faster processing and better use of computer power.
What this means in practice
- •For ai service operators: Increase throughput and utilization when running large language models for many simultaneous users by dynamically sharing resources across draft and verification stages.
- •For cloud infrastructure teams: Deploy flexible resource pools that optimize GPU use by decoupling and dynamically scheduling parts of language model inference pipelines.
Authors
Junhao He, Hongyang Du
Abstract
Speculative decoding accelerates Large Language Model (LLM) inference by using a lightweight draft model to propose candidate tokens for parallel verification by a target model. Drafting and verification, however, exhibit different service characteristics and favor different batch configurations, making fixed draft-target coupling inefficient under concurrent workloads. Existing distributed designs can physically separate the two stages, but often retain request or batch affinities that prevent their capacities from being shared globally. We present NebulaSD, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools. Such dynamic reassignment removes fixed worker locality, requiring request states to be made available at newly selected workers without introducing migration stalls. NebulaSD addresses this challenge through worker-triggered batch reconstruction and asynchronous KV-state preparation overlapped with model execution. We evaluate NebulaSD from both system and scaling perspectives, showing that dynamic pooling improves request-round processing rate by 50.4% over a physically disaggregated baseline and 72.6% over co-located execution on a four-GPU deployment while substantially increasing effective GPU utilization. Profile-driven simulations further show approximately proportional compute-side capacity scaling under idealized state movement.