Summary
People use reinforcement learning (RL) to make language models better, but it is unclear how RL changes the model internally. The authors studied what RL does inside the model and found that it mostly adjusts how often certain existing features appear, rather than creating completely new ones. These changes happen slowly, mostly near the end of the model's processing layers, focusing on formatting details like punctuation and answer separators. They also showed that replicating these adjusted features in the base model recovers most of the benefits of RL, implying RL hones what the model already knows rather than teaching it entirely new things.
What this means in practice
- •For language model developers: Track and target specific internal features that RL enhances to improve model capabilities more efficiently without retraining from scratch.
- •For ai safety engineers: Monitor subtle representation changes caused by RL to better understand and control model behavior shifts over training.
Authors
Lingheng Du, Yiming Tang, Xufeng Duan, Dianbo Liu
Abstract
Reinforcement learning (RL) is widely utilized in large language model training to improve targeted capabilities, yet how RL reshapes a model remains poorly understood. Prior attempts to explain how RL works largely offer behavioral perspectives, leaving open what RL gives a model at the representation level: can RL create genuinely novel features, and which existing features does it enhance or suppress? Recent developments in mechanistic interpretability suggest sparse autoencoders (SAEs) as a promising lens to decompose internal activations into human-interpretable features; however, they cannot be directly applied to tracking change across training. In this work, we introduce Fixed-SAE Track, a framework that trains one shared SAE per considered layer on activations pooled across the base model and all RL checkpoints, holding every feature direction fixed so that representation shifts are rigorously defined through the activations of interpretable SAE latents, including the detection of emerging novel features. Validated across multiple datasets and RL algorithms, we find that RL-induced drift is small, gradual, concept specific, and concentrated in late layers, mainly enhancing the sampling rates of a small set of ladder tokens, formatting scaffolding such as step breaks and answer delimiters, rather than reshaping problem content. Steering these features into the base model recovers around 80% of RL's performance gain, suggesting that RL primarily elicits capabilities the model already possesses, much as steering does. We further design a synthetic benchmark with features known by construction to test whether RL can instill genuinely novel features. We believe Fixed-SAE Track provides a principled approach to tracking representation shifts and offers representational evidence for understanding how reinforcement learning changes the inner representation of LLMs.