VSkipper improves speed of large language model serving by skipping layers
vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models process information by passing tokens through many layers, which uses a lot of computing power and time. Some methods skip processing certain layers for some tokens to save time, but they don’t always make the overall system faster because they are hard to fit into existing software. The authors introduce vSkipper, a software layer that fits these skipping methods into current systems efficiently, allowing real speed gains without hurting output quality. They tested vSkipper on popular language models and found it made serving responses faster and increased the number of requests handled at once.
What this means in practice
- •For cloud ai platform engineers: Integrate dynamic layer skipping into large language model serving to reduce latency and increase throughput without specialized code changes.
- •For real-time ai application developers: Use vSkipper to speed up AI model responses in systems requiring low latency, improving user experience during interactive tasks.
Authors
Wei Da, Yavuz Ferhatosmanoglu, Evangelia Kalyvianaki
Abstract
Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely on specialized generation loops and do not integrate with modern serving engines. As a result, fewer executed layers do not necessarily translate into lower serving latency: FlexiDepth skips 8 of Llama-3-8B's 32 layers on average, yet its standard generation loop decodes 14.6--21.0% more slowly than the base model. We present vSkipper, a virtualization layer that makes dynamic layer skippers pluggable in serving engines while preserving continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs. At each routed layer, vSkipper groups tokens by the skipper's decision and uses routed execution only when predicted to be profitable. We implement vSkipper in SGLang and evaluate the released FlexiDepth checkpoint against upstream SGLang under identical prompts, arrivals, output lengths, and launch settings. At the knee of upstream's load curve, vSkipper reduces mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH. Under saturation, it increases request throughput by 11.3% and 7.4%. Serving adds no statistically resolved quality loss beyond the checkpoint's own. Across synthetic skip policies, two Qwen3 skippers, and three GPUs, we demonstrate reuse without workload-specific tuning. To our knowledge, vSkipper is the first system to realize serving-efficiency gains from per-token interior layer skipping within a modern LLM serving engine. The code is open-sourced as an SGLang fork at https://github.com/AKafakA/sglang-vskipper/tree/vskipper-ref