Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents
2026-06-02 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors study how language model (LLM) agents can accidentally reveal secret credentials when mixing trusted information with untrusted content. They explore three defense methods: spotting credential access before the model outputs anything, using special fake tokens (honeytokens) to detect leaks, and tracking information leaked over many conversation turns. Their tests show these methods can identify risky prompts better than just checking the text output. However, their results are early and based on limited data and models, so more research is needed.
LLM agentscredential exfiltrationprompt injectionactivation probeshoneytokenssplit conformal predictioninformation flow trackingmulti-turn conversationwhite-box accesstext output filtering
Authors
Kargi Chauhan, Pratibha Revankar
Abstract
LLM agents often place sensitive credentials in the same context window as untrusted retrieved content, creating a direct path for indirect prompt injection to induce credential exfiltration. We study this failure mode through three complementary defenses. First, we ask whether activation probes can detect credential access before output tokens are emitted. Second, we construct honeytokens from format-specific character models and calibrate detection with split conformal prediction. Third, we treat multi-turn exfiltration as a cumulative information-flow problem and track an estimated leakage budget across conversation turns. In controlled experiments on open-weight models, activation features separate benign and credential-seeking prompts with high accuracy, including under held-out encoding transformations. In a small synthetic multi-turn suite, cumulative accounting detects attacks that per-turn detectors miss. These results are preliminary: the multi-turn benchmark is in-house and small, the activation method requires white-box access, and the information estimator provides a practical signal rather than a formal upper bound. Still, the results suggest that credential-exfiltration defenses should combine pre-output monitoring, calibrated canary detection, and temporal leakage accounting rather than relying only on text-level output filters.