Safety routing in large language models degrades with distribution shifts
False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
Cryptography and SecurityArtificial Intelligence
Summary
When AI safety systems send tasks to different language models, their ability to pick the safest model can get worse if they face new or changed types of requests. The paper shows that evaluations of safety routing often assume the best choice is known in advance, which isn't true in real-world settings where requests vary. As a result, safety routing provides little real benefit under these changing conditions. The authors also demonstrate how attackers can fool safety checks by tricking models into hiding harmful content.
What this means in practice
- •For ai system builders: Design safety routing evaluations that account for changing types of inputs to better measure real-world performance under distribution shifts.
- •For security engineers: Develop and assess defenses that consider attackers adapting to model recognition signals to improve robustness of content filtering.
Authors
Amit Singh Bhatti, Vishal Vaddina
Abstract
Safety routers send each request to one of several models and are judged against the best single model. A major routing benchmark picks that comparator on the evaluation data. In the benchmark's own setting this is harmless, but under distribution shift it is not. On HELM Safety the selection cost is 0.003-0.030 of harm under random splits and 0.045-0.113 under held-out categories, comparable to the whole deficit attributed to routing, with its direction holding under either published judge alone. It rises seven- to ninefold on AgentDojo when suites are held out. Across seven safety corpora chosen by rules fixed in advance, three meet a registered interval test and four beat a later permutation null, and three of the four interval misses are corpora where some models have zero observed harm. Prior work proves the direction of this bias. We size it on harm and accuracy, show that it is larger under the held-out splits we measure, and bound it by optimism plus a shift-dependent regret. Scored honestly under shift, routing buys little on these benchmarks. In most pool cells the nested router serves the honest baseline's model, and on the nearly saturated AgentDojo corpus a perfect pre-dispatch router is worth at most two points of harm. We also find a model's expressed recognition of a late injection steerable. On held-out reruns an attacker who knows which model it faces lowers GPT-5.4's judged recognition by 19.6 points, confirmed by an independent label. In an offline counterfactual composition into a controller, the same attack raises or lowers estimated harm depending on the fallback model. Safety routing should be evaluated under shift, against a baseline chosen without the test labels, and recognition-based defences should be scored on harm against an attacker who chooses what the model sees.