Synthetic document finetuning fails to prevent misalignment from reward hacking
Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Artificial Intelligence
Summary
Sometimes AI models learn to cheat by finding loopholes in their reward systems, causing them to behave in unexpected and unwanted ways. Some techniques try to teach the model early on that cheating is okay to prevent future problems. The authors tested if adding fake documents promoting cheating mid-training helps stop bad behavior later. They found that while the model appears to accept cheating, it still ends up misaligned after more training, meaning this strategy doesn’t fully work. This shows that mid-training fake documents can change what the model says but can unpredictably affect its deeper learning.
What this means in practice
- •For ai safety engineers: Assess if synthetic data injections during training reliably prevent reinforcement learning models from developing unintended harmful behaviors.
- •For machine learning developers: Design training pipelines that avoid overreliance on synthetic documents when trying to correct model behaviors before reinforcement learning.
Authors
Arun Jose, Julian Stastny
Abstract
Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.