Efficient routing method improves large language model cascades accuracy
Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
Computation and Language
Summary
Large language model cascades try to answer easy questions with smaller models and send harder ones to bigger models. The usual way to decide when to send questions up is based on how uncertain the small model looks. The authors found that this misses something important: switching to the big model actually helps only if it fixes an error, and hurts if it changes a correct answer to a wrong one. They created Signed Rescue Routing, which predicts both good and bad switches separately and chooses when to escalate based on the net benefit. This method improves accuracy while keeping compute costs low.
What this means in practice
- •For machine learning engineers: Optimize AI services by routing requests to larger or smaller models to improve accuracy while reducing compute cost.
- •For cloud infrastructure teams: Implement cost-aware model serving that escalates only valuable cases to bigger models, reducing operational expenses.
Authors
Zheyuan Wang, Siyu Li, Peiqiao Song, Sijia Chen, Qianqian Song, Qian Liu
Abstract
Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.