LLM routing improves accuracy and cuts cost without retraining
COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use
Artificial Intelligence
Summary
Choosing the best AI language model for a specific task can be expensive and tricky, especially when models and tasks change often. The authors propose COMPASS, a smart routing system that uses a language model to gather information and check answers before picking the best model. Their tests show COMPASS can be more accurate and cheaper than existing methods without needing special training. It also adapts well when new models are added or when used on different types of tasks.
What this means in practice
- •For ai service operators: Reduce costs and improve response accuracy by dynamically selecting the best AI model for user queries without retraining routing systems.
- •For software engineers: Build adaptive AI systems that automatically incorporate new language models and tasks for improved performance without manual updates.
- •For customer support platform developers: Enhance automated response systems by routing requests to the most effective language model based on task and cost profiling.$Commercial implications: Enables commercial customer support products to optimize AI usage for better accuracy and lower costs, distinguishing services in the market.
Authors
Amin Rezaei, Pouya Parsa
Abstract
LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at \$139.57, compared with RLCascadeRouter's 69.14% at \$180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at \$215, compared with 74.1% at \$500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.