SPLASH enables dynamic switching of attention layouts in large language model serving
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
Different ways of sharing work in large language models work better under different conditions, like how many users there are or how long the requests are. The authors noticed it’s possible to switch between these sharing methods while the model is still working, without stopping everything. They built SPLASH, a system that changes how the model splits its workload on the fly with very little delay. This leads to faster and more efficient model serving depending on what kind of requests are coming in.
What this means in practice
- •For ai infrastructure teams: Improve throughput of large language model servers by dynamically adjusting attention parallelism based on ongoing request characteristics.
- •For cloud service operators: Offer more efficient large language model serving services by supporting seamless parallel layout switching to handle variable loads and prompt lengths.$Commercial implications: Enables cloud providers to sell more flexible and cost-effective LLM serving solutions with better resource usage.
Authors
Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao
Abstract
No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel attention, and long prompts favor context parallelism. Reasoning, agentic, and RL-rollout workloads make a fixed choice untenable: a batch that begins as many short requests ends as a few very long ones, so the best layout changes while the same requests run. Serving engines nevertheless fix one layout at launch, because changing it has meant draining requests and restarting workers. We present SPLASH, a serving system that switches the parallel layout of attention while requests are running. It builds on one observation: modern attention, with few or no KV heads, decouples where a request's KV cache lives from how attention weights are sharded. This has two consequences. First, layouts differ only in who owns the weights and the cache, and most of that state already sits where the next layout needs it; SPLASH reuses it, moves the rest in the background of ongoing inference, and hands off at a batch boundary, making a switch nearly free: its median overhead is under 0.51% of the step it runs in. Second, the decoupling exposes a layout that existing engines lack: Decoupled Ownership Parallelism (DOP) shards attention weights as tensor parallelism does while keeping each request's cache on a single owner as data-parallel attention does. DOP replicates neither, offers 27-60% more KV capacity than data-parallel attention, and gives the scheduler a choice when KV memory limits admission. A transition-aware scheduler follows the best of the four layouts as load changes. On B200 GPUs serving GLM-5.3, SPLASH improves end-to-end serving throughput by 1.3-1.73x over fixed-layout deployments, and the same layout regimes appear with DeepSeek-V3.2 on H200 and GLM-5.3-Flash on DCU.