Multi-turn large model routing improves success with better training targets
Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing
Artificial Intelligence
Summary
Handling tasks that need multiple steps using different large language models (LLMs) can be costly and tricky. This paper studies when to switch between models during a task and how to train the system that decides the switch. The authors find that allowing a later switch doesn't always help and can increase costs. They show that training with improved, clearer targets helps the system pick better moments to switch models, improving success and saving costs. They tested their approach on several datasets and in a live system handling user queries.
What this means in practice
- •For llm system developers: Improve multi-turn task performance and reduce costs by using better supervision signals for routing decisions between models.
- •For customer support platform teams: Deploy smarter LLM routing to resolve user requests more efficiently by selecting optimal switch points between language models.$Commercial implications: This paper enables cost-effective, accurate multi-turn query handling that could power commercial customer support AI solutions.
Authors
Daniel Matos, Lu Liu, Yanfeng Zhang
Abstract
We study how initial model selection, later handoff, and training targets affect success and cost in LLM tasks that span multiple turns. We compare several router architectures using handoff curves, which record task success and cost as the handoff step changes. Our dataset contains 34,204 complete executions of three Qwen models across BFCL, MultiWOZ, and $τ$-retail, providing offline supervision for routing decisions. In our primary comparison, with Qwen3.5 35B-A3B as the starting model, allowing a later handoff does not establish an improvement in utility over initial selection on BFCL or MultiWOZ, and increases cost on BFCL across a range of cost weights. Changing the supervision targets improves router performance on both our dataset and TwinRouterBench. On our dataset, replacing noisy training success curves with monotone fits raises success by 0.2 to 4.1 percentage points at the same cost across the reported operating points. On TwinRouterBench, we construct targets that describe the model capability needed for the remaining work and difficult steps ahead. Compared with training on the benchmark's per-step labels, these targets raise trajectory pass scores by 7.8 percentage points and cost savings by 6.3 percentage points on the full static bank. A live deployment of the router trained with these targets resolves 79 of 100 instances at a router cost of 0.59 US dollars per instance.