Benchmark labels stepwise tool call attacks on language model agents
AgentDrift: A Step-Labeled Benchmark of Injection-Hijacked LLM Agent Trajectories
Cryptography and SecurityArtificial IntelligenceMachine Learning
Summary
Language model agents use tools to complete tasks, but bad actors can sneak in harmful instructions partway through their actions. The authors created AgentDrift, a large collection of example sequences where these attacks happen at specific steps, with clear labels showing when the attack starts and affects the agent. This lets developers better study and detect these sneaky attacks which are hard to spot by looking only at the whole activity. The dataset includes normal actions, successful attacks, failed attacks, and tricky cases that look like attacks but aren't.
What this means in practice
- •For security engineering teams: Train and test detection systems to identify the exact points and impact of injection attacks on language model agent tool-call sequences.
- •For software developers building ai assistants: Use the benchmark to evaluate and improve the robustness of AI assistants against prompt injection attacks during multi-step tool use.
Authors
Asif Pinjari, Mithun Paul Saint-Germain
Abstract
LLM agents complete tasks by issuing sequences of tool calls, and every observation they read is a channel through which an indirect prompt injection can enter. A successful injection has a characteristic shape when the trajectory is read in order: a benign prefix gives way to actions that serve the attacker rather than the user. Existing benchmarks measure whether such attacks succeed against live agents, and existing guard models judge a trace as a whole; no public corpus labels, step by step, where an injection enters a trajectory and which steps it corrupts. We present AgentDrift, a benchmark of 12,536 synthetic tool-call trajectories over five agent domains in which every one of the 71,024 steps carries one of four labels: benign, injection point, hijacked, or failed injection. The corpus contains 4,000 benign, 5,536 attacked, 1,500 failed-attack, and 1,500 hard-negative trajectories; attacked trajectories follow three compliance patterns whose label strings obey a stated regular grammar. Failed attacks carry an injection the agent resisted, and hard negatives carry legitimate content that resembles an attack, so a detector must separate attempt from success and deviation from novelty. Trajectories were generated by a single open model under category-specific protocols, enforced by a closed-vocabulary structural validator, screened by an LLM judge, and audited by hand on 1,200 trajectories; we show that the LLM judge was itself fooled by the hard negatives. A surface-feature logistic regression recovers only 55.4% of attacks (F1 0.647), including only 8.2% of partial hijacks and 23.1% of delayed executions, so nearly half of the attacks require modeling the behavioral sequence. We measure template concentration, attack-goal-family collapse, and world-identity leakage in the generated data, and release the corpus with its documentation under CC BY 4.0.