Semantic entropy helps decide when to call bigger language models
Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself
Artificial Intelligence
Summary
Sometimes smaller AI language models can answer questions well, but other times they make mistakes. The authors studied a way to predict when a small AI might mess up by looking at how much its possible answers disagree in meaning, called semantic entropy. They found this method can help decide when to ask a bigger AI for help, saving time and cost. However, they also warn about common tricks that can fool this prediction method, so they offer checks to avoid being misled.
What this means in practice
- •For ai system engineers: Improve AI services by choosing when to use smaller or larger models based on semantic disagreement signals to balance accuracy and cost.
- •For automated customer support teams: Reduce expensive calls to large AI models by routing difficult customer questions only when necessary using semantic entropy checks.
Authors
Ramin Pishehvar, Andrea Morandi, Mahesh Viswanathan
Abstract
Deciding when to escalate a query from a small language model to a larger one requires a cheap signal that predicts, before the large model is called, whether escalating would help. Semantic entropy, originally developed to detect hallucinations, is a natural candidate: it measures how much a model's sampled answers disagree in meaning, and high disagreement often signals an unreliable answer. We test it across three benchmarks and two model families. On GSM8K, with a small/large pair about twelve times apart in size, semantic entropy reliably distinguishes the small model's mistakes (AUROC 0.871) and improves routed accuracy over random escalation by up to nine points at matched cost. An earlier strong-looking result on a synthetic benchmark proved misleading: a simple rule based only on question difficulty, with no model involved, matched semantic entropy almost exactly. This paper's main contribution is a set of checks that catch this before it is reported as real. We show that scoring a cheap, question-only difficulty estimate alongside any signal reveals whether the signal adds real information or just tracks how hard a question looks; that two reasonable definitions of "escalation worked" can produce very different results on the same data; that a benchmark can leave almost no room for any signal to beat simply always using the large model; and that the true cost of live sampling can make routing more expensive than calling the large model directly. For a cheaper alternative that reuses cached past outcomes, we show how to predict whether it will work on a new dataset -- confirmed by correctly forecasting a collapse from AUROC 0.908 to chance level (0.518) ahead of time. We offer these as a general checklist for evaluating escalation signals.