Probe accuracy rises but steering repair varies with model size
Decodable In-Context State and Model Output Across Training
Machine Learning
Summary
This paper studies how well you can read a model's internal state to understand and fix its mistakes as it trains over time. The authors found that the ability to decode these internal signals improves as training progresses, but using this information to correct errors works better only in some models. They also show that just because you can decode error signals inside the model doesn't mean all important output information is preserved. Some questions remain about how these internal signals relate to what the model leaves out in its final answers.
What this means in practice
- •For machine learning engineers: Monitor model training by tracking probe accuracy to identify when internal signals become usable for error analysis.
- •For ai system maintainers: Use probe-guided interventions selectively on medium-sized models to improve model output quality during deployment.
Authors
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
Abstract
Prior work established that a probe can decode an in-context binding on model errors and that probe-guided steering can repair some of them. We follow probe accuracy, model output, and steering response across public pretraining and post-training checkpoints. Probe accuracy rises during Pythia pretraining, while probe-guided steering moves from negligible all-trial benefit to a larger benefit at two model sizes. Saved scores distinguish probe-correct errors with low and above-uniform model probability for the correct candidate. Oracle-target steering already repairs many early errors, but saved aggregates cannot separate target quality from intervention sensitivity. A held-out comparison of decoders trained on the final state or candidate logits finds no detected final-state advantage on late-checkpoint model errors. An information-theoretic counterexample explains why decodability on errors alone cannot establish discarded output information. The connection to downstream omissions remains open.