Targets reuse successful text attacks and improve AI defenses quickly
What Victims Take Away: Attack Reuse and Defense Revision after Prompt-Injection Attacks
Cryptography and SecurityHuman-Computer Interaction
Summary
People who are attacked in a game that uses language model defenses often copy the exact text of attacks that worked on them to attack others. The researchers found that players reuse attacks mainly if those attacks succeed, not just if they look promising. After being attacked successfully, players also change their defenses, which makes similar attacks less effective next time. Most reuse came from players already attacking, and about a quarter of the successful attacks were repeats of previous ones. The authors suggest ways for AI security tools to share attack information without giving away the full attack text.
What this means in practice
- •For ai security teams: Design attack reporting practices that limit copying of usable attack text while helping defenders improve protections.
- •For cybersecurity operations teams: Identify which successful text-based attacks are being reused to prioritize defense upgrades and monitor attacker behavior.
Authors
Zihan Chen, Lei Nico Zheng, Di Zhu
Abstract
Attack logs and red-team reports show targets the attacks used against them, to help them repair their defenses. Prompt injections against large language models (LLMs) are plain text, so such records also hand targets a working copy. Drawing on social learning theory and work on the victim-offender overlap, we study what targets do with that copy in Tensor Trust, an online game whose players defended LLM-guarded accounts, attacked one another's, and could read every attack they received. Using all 563,349 attack records, we compare each received attack with matched attacks that circulated among similar players at the same moment but never reached the target. Reuse mainly followed success: within a week, targets attacked other players with 10.0% of the attacks that had broken their defense, versus 0.03% of matched attacks (adjusted difference: 8.8-10.0 percentage points). Near-misses, whose replies began with "access granted" but that scored as failures, were reused about as rarely as failures: reuse tracked whether an attack paid off, not how its reply read, consistent with payoff-biased social learning. Breaches were also followed by defense changes, after which same-family attacks became 24.5 percentage points less successful against the target, relative to other defenders. Reuse came mainly from players already attacking, not newcomers. Previously received texts accounted for 23.2% of the successful hijacks we analyze. The attack history thus served both repair and reuse; we derive testable hypotheses for how AI security tools should disclose attacks, such as describing the technique without the full text.