LayerRoute speeds up large language models by skipping transformer layers
LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference
Computation and LanguageArtificial Intelligence
Summary
Large language models use many layers of computation, which can be slow. This paper introduces LayerRoute, a new way that decides for each input which layers can be skipped to save time. The method trains small additional parts that learn when to skip layers without reducing the model’s quality. The authors show LayerRoute speeds up the model slightly while keeping or improving its accuracy and that the skipping decisions truly depend on the input text.
What this means in practice
- •For machine learning engineers: Use LayerRoute to speed up inference of transformer-based language models with minimal loss and manageable tuning overhead.
- •For cloud service operators: Deploy LayerRoute to modestly reduce compute time for language model APIs, saving operational costs without degrading output quality.
Authors
Prateek Kumar Sikdar
Abstract
We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with a lightweight per-layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters), training both jointly under a gate-regularized language-modeling objective. Across 10 independently-seeded training runs, LayerRoute converges to an identical skip-pattern structure in every run - a consistent set of 9 middle layers (8-16) becomes skip-eligible in all 10 seeds - and delivers genuine, verified wallclock speedup in every run (1.02x-1.06x, mean 1.04x). Quality is preserved or improved in every configuration tested: joint LoRA adaptation yields a perplexity improvement over the unmodified backbone in all 10 seeds (mean delta = -1.16 and -1.11 across the two evaluation splits used). We further verify the router performs genuine, non-trivial per-input computation: gate decisions in skip-eligible layers change the actual skip/run outcome for 87-100% of held-out samples, confirming real input-dependent routing rather than a fixed pruning pattern. LayerRoute trains in under 7 minutes on a single A100 and adds negligible overhead beyond the routing decision itself. We report our full reproducibility methodology, including a systematic diagnostic investigation into what determines the router's per-input decisions, as part of this work.