Large language models leak secrets through output only access
Practical Secrets Extraction against Black-box LLMs
Cryptography and Security
Summary
Large language models used for coding sometimes remember secret keys or passwords from their training data. These secrets can be accidentally revealed even if you only see the model’s outputs and cannot access its internal workings. The authors created a method that carefully crafts questions and analyzes answers to uncover these hidden secrets from black-box models like OpenAI’s and Claude Code’s APIs. Their approach improves how well and how quickly these secrets can be found compared to earlier methods.
What this means in practice
- •For security teams: Detect and evaluate secret leakage risks from black-box language models used in automated coding tools.
- •For api providers: Improve design of LLM APIs by identifying ways secrets can be extracted via output-only queries.
Authors
Shiqian Zhao, Siwei Jiang, Xinfeng Li, Runyi Hu, Yandan Zheng, Congyu Guo, Tianwei Zhang, Anh Tuan Luu
Abstract
Large language models (LLMs) increasingly power autonomous coding agents such as Codex and Claude Code, yet their training corpora may contain confidential credentials exposed in public repositories or collected from private development artifacts, creating risks of memorization and subsequent leakage. Existing extraction audits, however, largely assume access to model weights or token probabilities. In this work, we present a black-box secret extraction framework for commercial, API-based LLMs under output-only access. It comprises (i) \emph{Cross-Validated Secret Knowledge Distillation}, which uses semantics-preserving prompt variants, response cross-validation, and provider-specific format filtering to distill secret-relevant behavior into a local white-box proxy; and (ii) \emph{Proxy-Guided Secret Extraction and Candidate Filtering}, which combines truncated top-$p$ sampling with local token entropy, $N$-gram frequency profiling, and provider-specific structural priors. On controlled API-key benchmarks, our framework improves recovery effectiveness and real-key rates over representative baselines while reducing extraction latency. A responsible real-world evaluation further recovers masked provider-specific credentials from three independently deployed black-box LLM systems spanning OpenAI and Claude Code, showing that memorized secrets can be exposed under output-only access.