Toolkit assesses hidden prompt attacks on AI assistants effectively
pikit: A Composable Toolkit for Indirect Prompt Injection Research and Evaluation
Cryptography and Security
Summary
Sometimes AI helpers get tricked by sneaky hidden instructions that change how they work without asking you. The authors created a toolkit called pikit that tests many ways these tricks can happen and how well different protections work. They found some defenses reduce the success of these tricks by a lot, but simple detectors can miss many attacks. This toolkit records everything during tests so people can study and improve AI security better.
What this means in practice
- •For ai security teams: Evaluate and improve defenses of AI assistants against hidden malicious instructions using the comprehensive pikit toolkit.
- •For software developers: Integrate testing of prompt injection threats into AI-enabled software development workflows with pikit’s customizable attack and defense modules.
Authors
Zonghao Ying, Xiangfan Wu, Bo Yang, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo
Abstract
Indirect prompt injection embeds malicious instructions within external content retrieved by LLM-based agents, altering target behavior without user authorization. We introduce pikit, a research toolkit designed to systematically evaluate these threats across three core dimensions: attacks (13 methods), channels (16 carriers across text and file modes), and defenses (9 prevention strategies and 3 offline detection baselines). Built on a decorator-based registry, pikit enables seamless extension of custom components without modifying core code, while a unified craft() API composes arbitrary attacks and channels in a single call. We evaluated the toolkit on the pi coding agent powered by an anonymized LLM in a production-like environment. Benchmarking 9 prevention strategies against high-risk attacks yields a 71.8\% relative reduction in attack success rate, with few\_shot\_warning and instruction\_hierarchy providing the strongest protection. Offline detection baselines achieve perfect precision but low recall, demonstrating that heuristic detectors complement rather than replace prompt-level defenses. To ensure reproducibility, each run automatically logs full prompts, agent event traces, session transcripts, and verdict records. Our code is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/Research/pikit.