Llm routing choices leak sensitive information despite content logging off
Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation
Cryptography and SecurityComputation and Language
Summary
Large language model (LLM) routers decide which AI model to use based on the content of requests, but these routing decisions can leak sensitive user information like medical or harassment topics, even when the main content isn’t logged. The authors studied 1.7 million real requests and found that the type of request influences which model is chosen, creating privacy risks. They tested ways to reduce this information leak, such as adjusting routing decisions to avoid bias, but some leakage remains difficult to eliminate. The findings highlight hidden privacy concerns in AI systems that pick models silently based on what users ask.
What this means in practice
- •For cloud platform operators: Improve privacy controls by detecting and mitigating sensitive information leaked through LLM routing logs without content logging enabled.
- •For ai service developers: Design LLM routing schemes that reduce sensitive-topic bias revealed by routing metadata, thereby protecting user privacy in multi-model systems.
Authors
Teng-Ruei Chen
Abstract
LLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off. We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts. We run pre-registered studies on 1.7 million real requests (WildChat-1M, LMSYS-Chat-1M) with two cost/quality routers and a domain router, survey eleven systems' logging, and test post-processing defenses. At matched length, the shift's direction depends on category and router. For RouteLLM at the 50% operating point, harassment and self-harm requests reach the strong model 19 points less often than comparable ones on prompts unseen in exploration, medical requests (exploratory: LLM labels failed their gate) 31 points less often on distinct prompts (both post hoc), and sexual requests 10 points more often (secondary); the other router's four are negative. Twenty RouteLLM decisions separate frequent medical askers with AUC 0.71, exploratory and below the pre-registered primary endpoint's 0.75 (domain router: 0.92, an upper estimate). Per-category length-matched parity with accurate labels removes the gap on real traffic, costing at most 0.2 accuracy points on RouterBench (post hoc), where routers' gaps on sensitive subjects (13-42 points, pre-registered) exceed those of an oracle routing by realized accuracy gain (1-11, post hoc). Per-conversation stickiness, per-user budget bands, and pooled parity fail, the last as categories' shifts differ in size or sign. A post hoc exact per-user rate hides only even-prefix strong counts and forfeits most self-assessed routing value; it preserves odd-position decisions, from which a post hoc log attack reaches AUC 0.73 after 20 RouteLLM requests (exploratory).