Detecting hidden leaks of AI model weights with ongoing safeguards
Anytime-valid detection of LLM weight exfiltration
Cryptography and Security
Summary
Large language models can secretly leak their internal data by making slightly unusual word choices when answering prompts. The authors show that simply checking one answer against a trusted copy isn’t reliable because normal randomness also causes differences. They create a method that looks at many answers over time, combining small clues to detect leaks while keeping false alarms under control. Their approach works even when attackers try to hide signals cleverly and adapts to different models.
What this means in practice
- •For ai platform engineers: Monitor deployed language model services continuously for subtle unauthorized data leaks in output tokens, reducing missed detections over time.
- •For cloud security teams: Detect covert exfiltration attempts from compromised AI servers by accumulating weak evidence while bounding false alarm rates for long-term surveillance.
Authors
Ines Ortega-Fernandez, Mateusz Kowalczyk, Keri Warr
Abstract
A compromised LLM inference server can leak model weights by encoding payload bits in otherwise plausible token choices. A replay of the same prompt in a trusted server can expose such deviations, but benign numerical nondeterminism also causes token mismatches. Patient attackers can therefore hide within normal variation unless evidence is combined across responses. We introduce a prompt-level e-process that calibrates whole-response mismatch events on trusted benign traffic and accumulates evidence sequentially while, under a calibration-transfer assumption, controlling the probability of any false alarm over an unbounded monitoring horizon. We evaluate it on four models against a seed-blind attack and a stronger seed-aware attack that hides payload bits only in near-ties to remain stealthy, analyzing the channel capacity vs detectability trade-off. Compared with a hard per-token alarm, the e-process combines weak evidence across responses while providing explicit anytime false-alarm control.