Large language model routing improves by learning reusable capabilities

Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

Artificial Intelligence

Summary

Large language model (LLM) routing decides which AI model best answers a question to balance quality and speed. Current routing systems learn only for specific tasks and need retraining when models or situations change. The authors propose RouteFM, which learns general routing skills from many environments, so it can handle new tasks and models without retraining. Their experiments show RouteFM works well even with few examples, outperforming other methods on new tests.

What this means in practice

  • For llm deployment engineers: Improve inference efficiency by routing queries to suitable large language models without retraining routers for new tasks or model pools.
  • For multi-ai system integrators: Combine diverse AI models effectively by using a single pretrained router that adapts to different candidate models and domains through behavioral context.

Authors

Guannan Lai, Han-Jia Ye

Abstract

Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--efficiency trade-off of LLM inference. Existing routers are typically learned through local fitting: a router is optimized for a particular query workload and candidate pool, and often requires additional supervision or retraining as the routing environment changes. We ask whether LLM routing can instead be approached from a foundation-model perspective, learning a reusable routing capability that generalizes across tasks, candidate models, and deployment conditions. To this end, we introduce RouteFM, which learns to characterize anonymous candidate models from behavioral context and infer their target-specific capabilities, rather than binding routing decisions to fixed model identities or a single environment. Through episodic pretraining across heterogeneous routing environments, this capability can be reused by a frozen router and adapted to new environments through context alone. Experiments demonstrate transfer across changes in domains, modalities, candidate pools, and context budgets, with the largest gains when behavioral evidence is limited. On MMR-Bench, which is excluded from pretraining, RouteFM outperforms the strongest baseline by 2.23 quality points with only eight observations per candidate. These results support moving LLM routing from repeated local fitting toward a pretrain once, route anywhere paradigm. Our code is publicly available at https://github.com/LAMDA-Model-Reuse/RouteFM.