System one decision models improve autonomous penetration testing tools
Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
Summary
Autonomous penetration testing tools use large language models to find and report security weaknesses but sometimes make mistakes like false alarms or overestimating risk. The authors explore using simpler, specialized decision models called System One that give clearer, more calibrated judgments to help with confirming findings and deciding priorities. They compare tests with and without these models on a web target and discuss future improvements and training methods for such models. Their work aims to make automated security testing more reliable and efficient.
What this means in practice
- •For security operations teams: Use calibrated lightweight classifiers to reduce false positives and prioritize real vulnerabilities in automated penetration testing workflows.
- •For cloud security developers: Integrate System One decision layers to improve trusted automated agents that perform reconnaissance and exploitation on cloud web targets.
Tested on one dataset.
Authors
Joas Antonio dos Santos Barbosa
Abstract
Autonomous penetration-testing harnesses use large language models (LLMs) for reconnaissance, exploitation, and reporting, but often rely on those same models to confirm findings, grade severity, and select agents. This can lead to false positives, inflated severity, and wasted compute. We examine how System One decision models, lightweight non-generative classifiers that return typed, calibrated verdicts, can support these decisions. We make five contributions. First, we define four decision points: finding adjudication, severity recalibration, agent pruning, and confirmation loops. Second, we present an exploratory NeuroSploit case study comparing one run with TypeSafe System One (Jev) and one without it against a web target containing 13 vulnerabilities. Differences in severity distribution, runtime, and grading by exposed data type motivate the architecture but do not establish statistical significance. Third, we review published specifications for Jev, Jev-Ultrafast, and the open-source Laya without assuming that results from other benchmarks transfer to penetration testing. Fourth, we discuss RLHF, RLAIF, RLCD, and RLHV as training approaches and their implications for trust in security decisions. Finally, we propose Rave, a domain-adapted System One model, and outline its training data, evaluation protocol, and potential effect on harness assurance.