New system detects and blocks secret prompt leaks in AI chatbots
Detecting and Mitigating Query-Based System Prompt Leakage Attacks
Cryptography and SecurityArtificial Intelligence
Summary
Some AI chatbots use secret instructions called system prompts to guide their answers safely, but hackers can try to trick the AI into revealing these secrets by asking many linked questions. The authors created AdvGuard, a tool that looks at patterns across multiple questions to spot these sneaky attempts early and stop them. AdvGuard works with many different AI models and learns from how questions relate to each other instead of looking at each one alone. Tests show it catches almost all attacks without falsely accusing normal users.
What this means in practice
- •For ai service providers: Protect AI chatbot backends from attackers extracting secret instructions to maintain integrity and safety in user interactions.
- •For cybersecurity teams: Detect coordinated attempts to leak confidential AI prompts across multiple queries and block such attack campaigns early.
Authors
Tushar Nayan, Pratik Poudel, Qian Lou, Yuede Ji, Jason Liu, Ruimin Sun
Abstract
Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.