Signed rescue routing improves efficiency of large language model cascades
Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
Computation and Language
Summary
Large language models (LLMs) use smaller models to answer easy questions and bigger models for harder ones. Usually, the decision to ask the bigger model relies on whether the smaller model is unsure or likely wrong. The authors found that this approach misses a key point: it only helps if the bigger model fixes errors, and it hurts when correct answers get replaced by mistakes. They propose Signed Rescue Routing, which predicts when using the bigger model will help or hurt and prioritizes requests where the gain is positive. This method works better than just guessing uncertainty and needs only small extra computing at deployment.
large language modelmodel cascaderoutinguncertaintyBayes optimalitysmall modelescalationQwen3accuracy-compute tradeoffentropy
Authors
Zheyuan Wang, Siyu Li, Peiqiao Song, Sijia Chen, Qianqian Song, Qian Liu
Abstract
Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.