Papers for

security engineers in ai service providers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Tool using agents often ignore hidden attacks without alerting users

Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

Abstract: Tool-using agents read untrusted external content, which exposes them to indirect prompt injection (IPI), where hidden instructions hijack the agent's tool calls. Benchmarks measure IPI security by the attack success rate (ASR), which treats every failed attack as a successful defense. To test this assumption, we audit 33,000 trajectories from eleven open-weight agents (2-14B parameters) on InjecAgent and AgentDojo. A label-blind LLM judge shows that most failed attacks are ignored (76-80%) rather than refused (17-18%). With identical model weights, changing only the system prompt shifts InjecAgent ASR from 0.2% to 43%. Our Attack-Task Outcome Matrix (ATOM) further shows that in 21.7% of AgentDojo episodes, the agent stayed safe but did not complete the user's task. A low ASR therefore mostly reflects agents silently ignoring injections, not refusing them. Evaluations should report how attacks fail and whether the user's task is completed, alongside ASR. We release the labeled corpus, judge prompt, and analysis code.

Fri 16 OctComputation and LanguageCryptography and Security
The gist
Some computer programs use external information to perform tasks, but hidden secret instructions in that information can trick them. The authors found that when these tricks fail, the programs usually ignore the bad instructions quietly instead of actively rejecting them. This means current ways of measuring how well the programs resist attacks can be misleading. The authors suggest reporting not just whether attacks succeed, but also how they fail and whether the task still got done.
Open → 2610.23254v1