CopRoute improves language model routing by considering failure dependence
CopRoute: Dependence-Aware Conformal Routing for Language Models
Machine Learning
Summary
When using multiple language models to answer questions, it's helpful if at least one model gets the answer right. Existing methods pick models based on their individual chances of success but ignore how their mistakes might overlap. The authors propose CopRoute, which picks models by seeing how many new questions each one answers correctly beyond what others already cover. This approach reduces the number of model calls needed while keeping the risk of missing all answers low.
What this means in practice
- •For machine learning engineers: Reduce the number of model calls for multi-model systems while maintaining reliability by employing dependence-aware routing.
- •For ai infrastructure teams: Design systems that select language models dynamically, minimizing computational costs without increasing the chance of failure.
Authors
Siva Rajesh Kasa, Santhosh Kumar Kasa, Karan Gupta
Abstract
Set-based language model routing aims to increase the coverage of the event where at least one model answers a query correctly while controlling the number of inference calls made. Existing risk-controlled routing policies bound the all-miss risk (the probability that every model in the selected set fails) by calibrating a single conformal threshold and applying it to a nested family of model sets. Because each model is scored in isolation, these policies ignore the dependence structure of model failures. Models that share training data or architecture tend to fail on the same queries, so two individually strong models can be jointly redundant while a weaker but complementary model covers the queries that the others miss. To account for this dependence, which existing risk-controlled policies ignore, we propose CopRoute, which orders models by how many new queries each one answers correctly beyond those already selected, decides per query how far down that order to go, and then picks the smallest cutoff that still meets the target risk. We show that CopRoute's ordering attains, on the design data, the best guarantee any efficient fixed-ordering algorithm can give for the average number of calls to the first correct model, and that a separate calibration step certifies the all-miss risk of the deployed sets. On RouterBench and RouterEval, CopRoute attains the fewest certified calls on nine of ten datasets at a target risk of 0.05 and on eight of ten at 0.10, making up to 79 percent fewer calls (median 24 percent) than the state-of-the-art risk-controlled policy at the same certified all-miss risk.