Conversational ai systems improve response speed with smarter model selection
RouterWorld: Scaling Conversational AI with Latency-Aware LLM Routing
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Conversational AI systems, like chatbots, need to quickly choose which language model to use to give good answers without waiting too long. The paper's authors found that previous tests didn’t focus enough on measuring how fast responses are or handle real-world challenges where not all information is known in advance. They created RouterWorld, a big new test that combines many conversations and measures both answer quality and response speed while simulating different learning scenarios. They used RouterWorld to test 13 methods for picking language models and found many methods focus too much on accuracy and ignore response delays.
What this means in practice
- •For chatbot developers: Use RouterWorld benchmark to test routing methods that balance response speed and quality under real-world constraints.
- •For cloud service operators: Optimize large AI model routing policies to reduce latency costs in multi-model conversational AI deployments.
Authors
Jinyang Li, Denis Emelin, Simone Tedeschi, Weiwei Cheng, Cristina Morariu, Haytham Assem, Jens Lehmann, Emine Yilmaz, Angus Addlesee
Abstract
Conversational AI systems built on Large Language Models (LLMs) require routing mechanisms that balance answer quality and response latency at scale. However, existing LLM routing benchmarks assume full observability of all candidate model outcomes, under-emphasize latency as a first-class metric, and fragment evaluation across heterogeneous protocols misaligned with real-world settings. We introduce RouterWorld, a comprehensive benchmark for LLM routing in conversational GenAI. RouterWorld unifies 371k single-turn responses and 76k multi-turn agentic trajectories, adopts natural user interactions with a standardized LLM-as-a-judge protocol, and decouples efficiency from transient pricing via hardware-grounded latency annotations. Crucially, RouterWorld formalizes routing under partial observability through two setups: Cold-Start, which restricts learning to sparse, static logs, and Voyager, which models continuous data streams where routing decisions shape future training distributions. Evaluating 13 routing baselines, we find that most routers over-emphasize accuracy while ignoring latency under partial observability. Code and benchmark data are publicly available.