Computer security tests can be tricked by attacker-controlled inputs
Forgeable Confirmation in Automated Computer Security Testing: Deterministic Rules versus AI Judges
Cryptography and Security
Summary
Automated computer security tools often decide if an attack worked based on certain clues. The authors found that many of these clues can be tricked if attackers control the data being checked. They created a way to predict when these clues are vulnerable to tricks and showed that using rules is often easier to fool than AI judges. Their results suggest better protections come from making sure attackers cannot control the decisive evidence. However, if the attacker controls the system being tested, this protection fails.
What this means in practice
- •For security engineers: Improve security testing pipelines by identifying and protecting against easily forged confirmation signals.
- •For security software developers: Design more robust automated vulnerability scanners by routing critical evidence through channels attackers cannot influence.
Authors
Akihisha Fujiyama, Niwase Shamim
Abstract
AI is increasingly used to automate computer security testing, and the tools must decide for themselves whether an attack succeeded. A finding that a deterministic rule confirms by observation is reported as fact, whereas one that an LLM judges exploitable is treated as an opinion. We ask whether the system under test can forge that confirmation. In offline security testing of a four-stage AI-assisted pipeline, nine of its fifteen confirmation mechanisms are forgeable, and forgeability is predicted entirely by whether the decision reads attacker-controlled data. We formalise this as an auditable attack surface and test it prospectively: on sixteen held-out mechanisms, predictions fixed before any attack separated forgeable from unforgeable mechanisms exactly, and across 12,203 mechanisms in public scanner templates the prediction was 99.9% accurate. Deterministic rules proved cheaper to forge than eight open-weight LLM judges, failing at 2% of attacker-controlled response content against a median of 50%. No implementation of one check was both robust and precise, and routing between a rule and an AI judge raised forgery to 99%. Moving the decisive evidence to a channel the attacker cannot write cuts attack success from 97% to 0%, and an escalate verdict recovers the sensitivity this costs. The protection fails when the scanned host is itself the adversary. The results bear on AI security agents and on benchmarks that score success by string matching.