Post-training changes how language models interpret and answer questions
Represented but misread: Post-training changes how language models read out what they represent
Computation and LanguageMachine Learning
Summary
Sometimes, language models seem to know something but answer in unexpected ways. The authors studied whether this happens because the model’s understanding changes after extra training or just how it turns that understanding into answers. They found that after further training, models often changed how they 'read out' what they knew, not what they actually represented inside. Also, a small amount of extra training data could shift this reading process significantly in different directions.
What this means in practice
- •For machine learning engineers: Improve fine-tuning strategies by monitoring how additional training changes the way model representations are translated into answers.
- •For ai safety teams: Detect and adjust cases where models represent correct information internally but produce misleading or inverted outputs after fine-tuning.
Authors
Jochen Laubrock
Abstract
Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($Δd' = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.