Large language models struggle to assess cyber threat levels accurately
Benchmarking LLMs for Threat Level Determination
Cryptography and Security
Summary
Determining how dangerous a cyber threat is can help keep computer systems safe. This paper tests several large language models (LLMs) to see if they can automatically figure out the threat level from online cyber threat information. The authors found that without extra training, the LLMs do a poor job. When they fine-tuned the models with specific examples, performance improved but still was not good enough for real-world use. This means more work is needed to make these AI tools reliable for cybersecurity tasks.
Large language modelsCyber threat intelligenceThreat level determinationZero-shot learningFine-tuningF1 scoreMISP OSINT feedsSupervised learningModel benchmarkingDomain adaptation
Authors
Han Wang, Murathan Kurfalı, Alfonso Iacovazzi
Abstract
The fast progress of large language models (LLMs) opens new opportunities in the management of cyber threat intelligence, but their reliability for operational tasks remains unclear. In this work, we benchmark LLMs on the task of threat level determination. First, we construct a curated dataset derived from publicly available MISP OSINT feeds. Next, we design a tailored prompt to systematically compare eight different LLMs under zero-shot conditions. Finally, we apply supervised fine-tuning on each model and perform a comparative analysis between baseline and fine-tuned versions. Our results show that zero-shot models achieve weak performance, with limited ability to correctly assign threat levels. Fine-tuned models, however, demonstrate substantial improvements, reaching F1 scores between 0.40 and 0.58 depending on the base architecture. Despite this progress, the performance is still low for practical deployment, highlighting the need for additional research on data quality, model adaptation, and domain-specific tuning.