Mid-training method improves large language models for cybersecurity tasks
MiST: Mid-Training LLMs for Cybersecurity
Cryptography and SecurityArtificial Intelligence
Summary
Cybersecurity uses very complicated language and involves important decisions, making it tough for AI language models to understand. The authors created MiST, a new approach where models are first adapted midway using a specially made set of cybersecurity texts before being fully trained. This process helps the models do better on cybersecurity tests compared to some previous models. Their method also makes it easier to train the models further for specific cybersecurity jobs.
What this means in practice
- •For cybersecurity engineers: Use mid-trained models to improve automated detection and analysis of cyber threats with more accurate domain-specific understanding.
- •For software developers: Build security tools that start from mid-trained models to speed up customization and improve effectiveness on particular cybersecurity tasks.
Authors
Oded Ovadia, Elad Ben Zaken, Elad Guttman, Orly Moreno Kadosh
Abstract
Cybersecurity combines high-stakes analysis with complex technical language, making it an impactful and challenging domain for LLMs. We present MiST (Mid-trained Security Transformer), a suite of 8B and 32B models that achieve strong performance on public cybersecurity benchmarks. We use mid-training as an intermediate adaptation stage between general pre-training and cybersecurity training. Rather than performing continual pre-training over large volumes of raw domain text, we curate a compact, expert-vetted seed corpus, and transform it into high-quality domain-specific synthetic training data. The final MiST checkpoints improve mean cybersecurity accuracy by +13.1 and +8.6 absolute percentage points over the corresponding Qwen baselines for 8B and 32B, respectively, corresponding to relative gains of +27.0% and +15.8%. Ablation results further show that these cybersecurity gains arise in the mid-training and supervised fine-tuning stages through a combination of the synthetic data generation flows. Furthermore, we show that MiST provides a stronger initialization for downstream task-specific fine-tuning adaptation and reinforcement learning.