Large language models struggle to assess cyber threat levels accurately
Benchmarking LLMs for Threat Level Determination
Cryptography and Security
Summary
Cybersecurity teams need to know how serious cyber threats are to respond properly. The authors tested several large language models (LLMs) to see if they could automatically judge threat levels from open data. They found that without special training, these models perform poorly at this task. After tuning the models with some examples, performance improved but still wasn’t high enough for reliable use in real settings. This shows more work is needed before LLMs can help with threat level determination in cybersecurity.
What this means in practice
- •For cybersecurity analysts: Improve cyber threat prioritization by using fine-tuned models to assist in threat level classification from open-source intelligence feeds.
- •For security operations teams: Incorporate tuned language models to speed up initial triage of threat reports, reducing manual workload despite imperfect accuracy.
Authors
Han Wang, Murathan Kurfalı, Alfonso Iacovazzi
Abstract
The fast progress of large language models (LLMs) opens new opportunities in the management of cyber threat intelligence, but their reliability for operational tasks remains unclear. In this work, we benchmark LLMs on the task of threat level determination. First, we construct a curated dataset derived from publicly available MISP OSINT feeds. Next, we design a tailored prompt to systematically compare eight different LLMs under zero-shot conditions. Finally, we apply supervised fine-tuning on each model and perform a comparative analysis between baseline and fine-tuned versions. Our results show that zero-shot models achieve weak performance, with limited ability to correctly assign threat levels. Fine-tuned models, however, demonstrate substantial improvements, reaching F1 scores between 0.40 and 0.58 depending on the base architecture. Despite this progress, the performance is still low for practical deployment, highlighting the need for additional research on data quality, model adaptation, and domain-specific tuning.