AI coding agents exposed to poisoned tests can self-insert security flaws
Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks
Cryptography and SecurityArtificial Intelligence
Summary
Old computer viruses could hide in compilers and keep coming back even when the code looked clean. The authors show that modern AI programs that write and improve their own code can be tricked in a similar way by giving them bad test examples. These poisoned tests cause the AI to add hidden security problems in future versions, like turning off website certificate checks. The problem can persist even if the AI later works only on good tests. The authors suggest AI coding tools need better defenses against these sneak attacks.
What this means in practice
- •For software security teams: Detect and prevent supply-chain attacks on AI coding agents by monitoring for poisoned benchmarks that induce insecure coding behaviors.
- •For ai system developers: Design self-modifying AI coding agents with better safeguards against adversarial training inputs that cause harmful code generation.
Authors
Franziska Roesner, Tadayoshi Kohno
Abstract
Thompson's "Reflections on Trusting Trust" showed that a compiler can be poisoned to reinsert its own backdoor, so that even recompiling clean source reproduces the Trojan. Today, substantial coding work is done by AI coding agents -- and increasingly, those agents generate new versions of themselves. We reconsider Thompson's attack when the "compiler" is a self-modifying coding agent. Can an adversary supply poisoned benchmarks to the agent's self-evaluation and self-improvement process to induce future versions of the agent to write vulnerable code on clean, held-out tasks? We instantiate this attack against three recently proposed self-modifying coding agents: the Darwin Gödel Machine (with our experimental modifications), the Self-Improving Coding Agent, and Hyperagents (both substantively unmodified). We demonstrate successful proofs-of-concept: for example, with Hyperagents powered by Sonnet 4.5, our poisoned benchmark leads the agent to self-evolve instructions that disable HTTPS certificate validation on neutral URL-fetching tasks. From our experiments, we distill properties of the vulnerability, benchmark, model, and agent scaffolding that are sufficient to enable a benchmark poisoning attack. Moreover, we show that contamination often persists even when a poisoned agent is subsequently evolved against clean benchmarks. We discuss defensive directions and argue that self-modifying coding agents must be designed to be more resilient to such attacks.