Large language models help label cybersecurity tweets for training AI
LLM-Based Weak Annotation for Cybersecurity Tweet Classification: A Teacher-Student Framework
Computation and Language
Summary
Cybersecurity experts need to quickly spot new threats from social media posts, but there are too many tweets for humans to read and label. The authors show that advanced language models, called large language models (LLMs), can automatically assign useful labels to unlabeled cybersecurity tweets. Using these labels as training data, they teach specialized AI models to classify tweets almost as well as if humans had labeled all the data. This method can save expert time when creating training data for cybersecurity tweet analysis.
What this means in practice
- •For cybersecurity teams: Automatically generate labeled datasets to train models that detect cyber threats from social media posts without extensive human annotation.
- •For social media monitoring teams: Create domain-specific AI classifiers for cybersecurity content on social platforms using weakly labeled data to improve threat detection.
Authors
Ferry Afit Kurniawan, Kazushi Okamoto, Koki Karube, Kei Harada, Atsushi Shibata
Abstract
Cyber threat intelligence (CTI) relies on the timely identification of emerging threats, enabling organizations and security analysts to support situational awareness and decision-making. Social media platforms such as X (formerly Twitter) have become valuable CTI sources because vulnerabilities and attack activities are often discussed there before appearing in formal reports. However, the rapid growth of cybersecurity discussions on social media has created more content than domain experts can realistically review and annotate, making labeled data a major bottleneck for developing supervised learning approaches. While large language models (LLMs) have shown promise as weak annotators in other domains, their effectiveness for cybersecurity tweet classification and downstream training of domain-specific pre-trained language models (PLMs) remains underexplored. To address this challenge, we propose a Teacher-Student weak supervision framework in which a selected LLM serves as a Teacher to generate weak labels for unlabeled cybersecurity tweets, enabling the training of cybersecurity-specific PLMs without large-scale manual annotation. Experimental results show that PLMs trained solely on LLM-generated labels consistently achieve F1 scores above 0.92, remaining within 3 percentage points of fully human-supervised models across all evaluated architectures. These findings suggest that LLM-based weak annotation offers a practical alternative for constructing cybersecurity-specific training data when expert annotation resources are limited.