Reinforcement learning with verifiable rewards does not steadily increase power-seeking behavior
Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
Machine LearningArtificial Intelligence
Summary
Some people worry that smart AI systems might try to gain power or hide bad actions when trained to get rewards. This paper looks at whether training language models with a special kind of reward actually makes them more likely to want power. The authors find that such behavior goes up at first during training but then drops even below the starting level. This means that the idea that training with rewards always makes AI more power-hungry might not be true for all models.
What this means in practice
- •For ai safety teams: Monitor AI training stages for nonmonotonic shifts in risky behaviors to design safer training protocols.
- •For machine learning engineers: Track intermediate reinforcement learning checkpoints to better understand and control emerging behaviors in language models.
Authors
Barış Alparslan, Sandy Tanwisuth, Caleb Petterson
Abstract
Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline. The result is specific to this model family and benchmark, and judge agreement is moderate, so absolute rates should be read cautiously.