AI summaryⓘ
The authors studied a method called Gumbel-based inference verification that tries to stop secret information leaks from large language models (LLMs) by only accepting token choices that look like normal random variations. They found this method works well if the attacker just listens passively, causing a big slowdown for hidden messaging. However, if the attacker controls the inputs to the model and uses weird, broken prompts, the defense becomes weaker and more data can be leaked. Their tests on several models showed this active strategy roughly doubles leaked information and reduces the defense's effectiveness. They suggest defenses need to adjust dynamically based on how unpredictable the model's outputs are, rather than relying on fixed thresholds set using normal prompts.
Gumbel-based inference verificationlarge language modelsweight exfiltrationtoken entropyadversarial promptssteganographycovert channelinstruction tuningGPU nondeterminismdynamic thresholding
Abstract
Gumbel-based inference verification bounds LLM weight exfiltration by only forgiving token choices that plausibly arise from honest GPU nondeterminism, reporting a >200x slowdown for a steganographic adversary under benign prompt traffic. This bound assumes a passive attacker; we show it degrades sharply against an adversary who instead controls the prompt distribution. Because the verifier's admissible-token-set size is driven by the model's own output entropy, prompts engineered to break grammatical and sub-word structure -- rather than benign conversational traffic -- widen that set and open a materially larger covert channel. Across six instruction-tuned models spanning 1B to 32B parameters and three random seeds, our strongest attack (character- and script-level disruption) roughly doubles bits leaked per token relative to benign prompts, cutting the slowdown factor to 60x - 118x. These results indicate that static, benign-traffic-calibrated thresholds are insufficient for this defense, and that jitter-forgiveness thresholds should instead be calibrated dynamically against local token entropy.