Local llms improve alert triage by separating probe choice from stopping decision
HESP: Separating What to Probe from When to Stop in Local LLM Alert-Triage Agents
Cryptography and SecurityArtificial Intelligence
Summary
Security teams get overwhelmed by too many alerts to check manually, and small language models running locally often struggle to investigate effectively. The authors developed HESP, a system that takes charge of deciding what to investigate and when to stop, instead of leaving those choices to the language model. By doing this, HESP helps smaller models finish their investigations reliably and avoid missing real problems. This approach works entirely on local computers, so it fits situations where data cannot be shared externally.
What this means in practice
- •For security operations teams: Automate triage of security alerts using small local language models with controlled probe selection and stopping rules to reduce missed cases and non-convergence.
- •For enterprise it departments: Deploy local alert investigation systems that do not require sending data outside the premises, meeting strict data privacy and compliance requirements.
Authors
Zhuowen Liu, Zhixuan Wang
Abstract
Security operations centers receive far more alerts than analysts can investigate, and organizations that cannot send their telemetry to hosted models must automate triage with small open-weight LLMs on their own hardware. Current LLM agents leave the investigation procedure to the model, and small local models fail at it: they probe without converging, never commit to a verdict, or dismiss real attacks. In this paper, we present HESP, a controller that holds the investigation procedure outside the model. HESP keeps a ledger of competing explanations, selects read-only probes by expected information gain per cost, accepts only verdicts backed by current evidence, can end an investigation itself, and journals every prediction before its observation. We evaluated HESP in four pre-registered studies with five open-weight models from two families (7B to 72B), totalling 7,272 audited episodes in a controlled triage environment. With likelihood tables counted from LLM-free runs, HESP lifts Qwen2.5-7B from 0.125 to 1.000 verified completion, matching oracle tables. The information-gain ranking adds +0.26 to +0.35 on every model that concludes, and a controller-side stop lifts Llama-3.1-8B, which never concludes on its own, from 0 to 0.917. What to probe and when to stop are therefore separate failures, and different small models exhibit different ones. Because HESP and its planner run entirely on local hardware, it suits environments where telemetry cannot leave the premises. We release all code, protocols, and episode journals at https://github.com/lzwhehe/HESP.