Llm agents show gaps investigating cyberattacks with varying logs
APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry
Cryptography and SecurityArtificial Intelligence
Summary
Investigating complex cyberattacks relies on analyzing computer logs, but these logs can vary in type and detail. The authors created a test called APTInvestBench that checks how well AI agents find evidence of attacks when different types of logs are available. They found that although AI agents often find important attack details, their ability to properly support these findings with evidence can drop if the quality or type of logs changes. This work helps identify when and why AI tools might fail in real-world security investigations.
What this means in practice
- •For security operation teams: Test and improve AI tools that automate cyberattack investigations under different logging setups.
- •For cybersecurity platform developers: Develop more reliable AI-driven investigation assistants by addressing evidence support gaps revealed under diverse telemetry.
Authors
Yu Wang, Shuhao Li, Tao Yin, Ziyang Li, Xueying Zhao, Peishuai Sun, Jiang Xie
Abstract
Large language model (LLM) agents could help security operations centers (SOCs) investigate advanced persistent threats (APTs) by turning weak leads into evidence for intrusion scoping and response. Yet success under one telemetry setting does not establish robustness to changes in log collection, retention, or sampling. We introduce APTInvestBench, a benchmark for evaluating cross-telemetry robustness in autonomous APT investigation. It comprises 370 cases across seven SOC-inspired conditions, derived from 56 report-informed attack reconstructions with 16.4 million log records. Agents investigate unverified leads and submit reports with record-level citations. Fixed action-level support requirements track sufficient evidence across available logs, query returns, and formal citations, separating telemetry limitations from acquisition and reporting gaps. Across eleven LLMs, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations support only 25.0%. More importantly, aggregate coverage can conceal substantial instability: from Full to endpoint-only telemetry, coverage declines by only 1.6 percentage points, yet 35.5% of previously covered actions lose sufficient citation support despite remaining recoverable. Across four frameworks, such losses persist even when registered supporting records remain unchanged. APTInvestBench provides reusable investigation environments and diagnostic evaluation for identifying these gaps and developing more reliable defensive agents.