History aware routing improves efficiency in large language model processing
Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
Artificial IntelligenceComputation and Language
Summary
Large language models process text by passing information through many layers, which can be slow and costly. Usually, the system decides at each layer whether to skip it based only on the current information, ignoring how previous decisions affect later ones. The authors propose a method called HeRo that remembers past routing choices, allowing smarter decisions about which layers to use. Testing shows that HeRo keeps or improves performance while skipping a significant portion of the work, especially helping in tasks like code writing and complex reasoning.
large language modelsdynamic routinglayer skippinginference costhidden staterouter memorylinear attentionfeedforward network (FFN)model efficiencytoken-wise processing
Authors
Hongjin Lin, Wentao Wan, Keze Wang
Abstract
Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the current hidden state which is a formulation that overlooks the sequential, path-dependent nature of routing across depth: earlier decisions shape the representations seen by downstream routers, and the layer-usage objective couples all decisions jointly. We propose History-Aware Routing (HeRo), a dynamic routing framework that resolves this mismatch by introducing a router memory mechanism to maintain an explicit routing state across model depth. The memory is constructed via linear attention, incrementally aggregating preceding routing scores and their induced residual updates into a compact history representation. At each routed layer, the router conditions jointly on this accumulated state and the current hidden representation to select the executed branch. Instantiated for token-wise FFN routing, HeRo trains only lightweight routers and adapters on a frozen backbone, requiring no modification to pretrained parameters. Across Llama 3.1-8B, Llama 2-7B, and Llama 2-13B, HeRo consistently achieves the highest aggregate performance retention among ten baselines. On Llama 3.1-8B, it bypasses 26.87% of model parameters while achieving 100.24% of dense model performance across seven benchmarks, and retains 97.01% while bypassing 38.82% of model parameters under a tighter computation budget. Ablation studies confirm that removing routing history consistently degrades performance, most notably on multistep reasoning and code generation, validating that explicit routing memory enables more accurate and adaptive dynamic routing than solely conditioning on hidden state.