TokenRouter improves decoding speed for token-level large language model routing
TokenRouter: Efficient Serving System for Token-Level LLM Routing
Computation and Language
Summary
Running large language models can be slow and costly, especially when switching between different models for each word generated. The authors identified that existing systems are not efficient enough for this fine-grained switching, causing delays and complexity. They built TokenRouter, a system that manages requests cleverly and batches work efficiently for different models. This system achieves much faster processing speeds across various scenarios compared to previous methods.
What this means in practice
- •For cloud infrastructure teams: Increase efficiency when serving large language model inference by enabling faster token-by-token routing across multiple models in distributed systems.
- •For real-time translation developers: Deliver quicker language translation by dynamically routing token-level computations to suitable language models without slowing down response times.
Authors
Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang
Abstract
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.