Layer sensitivity patterns evolve during language model training
Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining
Machine LearningComputation and Language
Summary
Language models are made of layers whose roles change as the model learns over time. The authors studied how response patterns to changes in individual layers shift throughout training. They found that while the order of these layer effects stays recognizable, the strength of responses moves and varies. This means the layers' importance is not fixed but evolves, and studying just one snapshot during training might miss this complexity.
What this means in practice
- •For language model engineers: Improve model analysis methods by accounting for changing layer sensitivity during pretraining to better understand model internals.
- •For machine learning platform teams: Design training monitoring tools that track evolution of layer responses to identify transient vs persistent model behaviors.
Authors
Shengye Tao, Yinzhu Cheng, Haihua Xie
Abstract
Layer interventions are widely used to probe the internal organization of language models, yet most analyses examine a single training checkpoint even though model representations and computations evolve throughout pretraining. This leaves open which depth-dependent intervention responses reflect persistent organization and which are transient consequences of training. We study this question using single-block identity bypass on fixed teacher-forced contexts across five released trajectories and 11 model-domain combinations. We find that block-bypass responses retain recognizable depth ordering while their magnitudes redistribute: nearby checkpoints preserve stronger rank correspondence than distant ones, and large changes concentrate at positions that recur across text samples and transfer across evaluation domains. Controlled experiments further show that changes in the natural bypass effect cannot be reduced to a single downstream sensitivity: in replicated Pythia runs, local missing-update magnitude grows while the pooled matched downstream response decreases, whereas OLMo-2 7B exhibits a different balance. These matched responses also depend on perturbation strength and direction, without identifying targeted compensation. Together, our results show that longitudinal layer sensitivity is structured but not static, and that single-checkpoint intervention responses should be interpreted in the context of how the underlying perturbation pathway evolves during training.