Safety training improves large language models without losing skill
Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability
Computation and Language
Summary
Large language models can sometimes give unsafe or harmful answers. Usually, safety training is done after the model is already trained, which can reduce how well the model can do other tasks. The authors suggest adding a safety-focused training step in the middle of training, which helps models get safer without losing their general skills. This approach also keeps the model safer even after later training without safety data. Overall, safety and general reasoning can be learned together to make safer and still useful language models.
What this means in practice
- •For ai development teams: Improve the safety of language models during training without sacrificing their general abilities.
- •For machine learning engineers: Develop language models that maintain safety features even after further fine-tuning with general data.
Authors
Zhengyue Zhao, Liwei Jiang, Yejin Choi, Chaowei Xiao1
Abstract
Safety alignment for large language models (LLMs) is primarily performed during post-training, while recent work has also begun incorporating safety data into pre-training. Yet safety can degrade when later training no longer includes safety data, and gains in safety often come at the cost of general capability. This tension suggests that safety and general reasoning should not be treated as separate training goals. We instead view safety as an LLM capability that can be learned through a reasoning structure shared with general tasks. Both require the model to understand intent or task goals, apply relevant knowledge, and reason about potential consequences. We introduce safety mid-training, a stage between pre-training and post-training that jointly develops safety and general reasoning through this shared structure. Given safety and utility prompts, the base model generates structured documents and reasoning traces that are then used for mid-training. Safety mid-training reduces average attack success rate (ASR) by 72% to 81% relative to Qwen3-1.7B, 4B, and 8B base models, while also improving general capability. The benefits persist through subsequent post-training. After general-only SFT, mid-trained models largely retain their safety, achieving lower ASR than the baselines without sacrificing general capability. They also retain safety under general-only GRPO without safety rewards, while subsequent RL reaches a better safety-utility frontier when initialized from the mid-trained models. Together, these results suggest that learning safety and general reasoning within a shared mid-training stage can reduce the safety-utility trade-off and make safety more robust to subsequent training.