Papers for

developer teams building ai assistants

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Detectors struggle to spot rare attacks and unseen tools in hijacked AI agents

What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer

Abstract: Detectors can flag hijacked large language model (LLM) agents from the traces of the tools they call. In the eleven evaluations we audited, attacks or anomalies make up 16% to 86% of the test samples, no evaluation varies that share, and six report no test on data from a second benchmark or source. We ask what such evaluations hide. On the AgentDrift and AgentDojo benchmarks we test trained detectors that read tool identities, behavior, call sequences, step embeddings or text, and a released LLM action monitor, AlignmentCheck, with thresholds fixed on training data or set by the monitor's released decision rule. First, at a 2% attack rate a detector with an area under the receiver operating characteristic curve (AUROC) of 0.879 reaches a precision of 0.076, and the mix of negatives alone moves that precision between 0.039 and 0.117. Second, as tool coverage, the share of test tool calls whose tool was seen in training, falls, detector rankings cross near 50% coverage and reverse below it, because unseen tools map to inputs that training never reached. Dropout on tool inputs largely removes the reversal. Third, no detector trained on AgentDrift ranks AgentDojo traces better than chance (AUROC 0.47 to 0.52), including one that reaches 0.996 within AgentDojo. The monitor, trained on neither, reaches 0.930, but with its open backbone its released decision rule would still mostly raise false alerts at a 2% attack rate (precision 0.096). We turn these findings into a checklist for evaluating trace detectors.

Fri 30 OctCryptography and SecurityMachine Learning
The gist
Detecting when AI assistants have been hijacked can rely on spotting unusual patterns in the tools they use. The authors found that standard tests give detectors an easier job by including many attacks and familiar tools, which hides real-world challenges. When attacks are rare or tools are new, detectors often fail or make many false alarms. The authors show that detectors trained on one dataset do poorly on another and propose guidelines for better testing.
Open → 2610.41803v1

Tool using agents often ignore hidden attacks without alerting users

Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

Abstract: Tool-using agents read untrusted external content, which exposes them to indirect prompt injection (IPI), where hidden instructions hijack the agent's tool calls. Benchmarks measure IPI security by the attack success rate (ASR), which treats every failed attack as a successful defense. To test this assumption, we audit 33,000 trajectories from eleven open-weight agents (2-14B parameters) on InjecAgent and AgentDojo. A label-blind LLM judge shows that most failed attacks are ignored (76-80%) rather than refused (17-18%). With identical model weights, changing only the system prompt shifts InjecAgent ASR from 0.2% to 43%. Our Attack-Task Outcome Matrix (ATOM) further shows that in 21.7% of AgentDojo episodes, the agent stayed safe but did not complete the user's task. A low ASR therefore mostly reflects agents silently ignoring injections, not refusing them. Evaluations should report how attacks fail and whether the user's task is completed, alongside ASR. We release the labeled corpus, judge prompt, and analysis code.

Fri 16 OctComputation and LanguageCryptography and Security
The gist
Some computer programs use external information to perform tasks, but hidden secret instructions in that information can trick them. The authors found that when these tricks fail, the programs usually ignore the bad instructions quietly instead of actively rejecting them. This means current ways of measuring how well the programs resist attacks can be misleading. The authors suggest reporting not just whether attacks succeed, but also how they fail and whether the task still got done.
Open → 2610.23254v1