Alignment midtraining shows limited success steering AI behavior
Stress-testing Alignment Midtraining
Computation and LanguageArtificial Intelligence
Summary
AI models need to behave well in many different situations, but it’s hard to check all possible cases after training. The authors studied a method called alignment midtraining, which tries to improve behavior by continuing training on special documents partway through learning. They found that while midtraining can influence the model’s goals in simple cases, even a small amount of conflicting data during later training can undo these effects. Also, the model only reliably learns rules if examples of those rules appear in both midtraining or later training data. Overall, the authors conclude there isn’t enough evidence to say midtraining solves key AI alignment challenges.
What this means in practice
- •For machine learning engineers: Evaluate midtraining steps carefully to avoid conflict with later fine-tuning data in AI behavior tuning pipelines.
- •For ai safety teams: Assess rule coverage in training data carefully to ensure intended behavioral constraints are learned by large models.
Authors
Sid Baines, Jonathan Bostock, Maria Angelica Martinez, Andrew Draganov, David Africa, Daniel Tan
Abstract
When aligning frontier models through post-training techniques, it is not possible to directly demonstrate all of the behaviours we want a model to exhibit in all possible deployment environments; our model must generalise outside of the post-training distribution. One proposed solution is alignment midtraining (AMT), which continues pretraining on large volumes of alignment-relevant documents to encourage generalisation in later stages of training. Despite the prominence of AMT as an alignment approach, there is limited public evidence for its effectiveness. To resolve this, we identify several assumptions around midtraining and evaluate them across scale: up to 110 billion-parameter models and 1 billion midtraining tokens. For instance, we study a scenario where post-training data is ambiguous between two possible motivations. We find that midtraining can steer the model's motivation in simple versions of this setting. However, the presence of a tiny fraction of finetuning data which suggests a competing motivation erases the effects of AMT. We also study scenarios in which we want an AI to follow a number of rules, but only demonstrate a subset of them. We find that demonstrations must be present either in midtraining or post-training datasets for these rules to be robustly learned. Based on these and other findings, we do not believe that there is sufficient public evidence for us to confidently state that midtraining can address the core difficulties inherent in aligning powerful AI systems.