Context attacks can trigger wrong actions despite typed decision limits
Poisoned into Action: Context Attacks on Typed Decision Workflows
Cryptography and SecurityMachine Learning
Summary
Some computer systems use strict rules to limit what actions can be taken automatically, but attackers can still find ways to trick these systems into doing harmful things. The authors created a method to track how misleading information influences decisions and their effects, including how defenses might still allow some correct actions. They tested their method on a specific automated system and found that some attacks could cause security problems, while certain checks reduce these risks but also block some correct actions. Their findings help understand when and how these poisoning attacks succeed or get stopped, though more study is needed.
What this means in practice
- •For automation system developers: Detect and analyze how malicious inputs affect decision workflows to improve defense designs in typed decision interfaces.
- •For security engineers: Evaluate the effectiveness of state-change protections against context poisoning when integrating AI decision models with automated actions.
Authors
Chenkai Wu, Xing Chang, Jun Yan
Abstract
Typed decision interfaces restrict models to predefined actions, yet schema-valid choices can still trigger policy-violating state changes. Evaluating context poisoning thus requires connecting model decisions to execution outcomes while accounting for legitimate automation retained by defenses. We introduce an execution-aware protocol tracing poisoned evidence through retrieval, model choice, automated action, and persisted state. We evaluate it with 1,659 fresh calls to Jev-1.13.0 across 20 development tasks in two synthetic workflows, using three repetitions per condition. Under natural retrieval, a standalone payload reaches all 60 trials and produces five security-violating executions, while a carrier-payload pair exposes the payload in 57 trials and produces one violation. An initial source-exclusion gate blocks the pair's observed violation but also withholds ten otherwise correct attacked decisions and sixteen correct clean decisions. Correcting its empty-scope behavior restores twelve clean decisions without additional model calls. Factorial controls show that adding correct evidence reduces wrong executions from 119/120 to 55/120 in the semantic arm and from 60/120 to zero in the neutral arm, suggesting that much of the improvement reflects general evidence value rather than attack-specific protection. Three prespecified contrasts are not statistically supported after Holm correction, so the findings remain exploratory. The protocol provides an auditable framework for studying which poisoning-induced errors reach execution and how much correct automation defenses preserve. Code is available at https://github.com/WuXixiong/jev-poisoning-bench.