Transformers encode partner expertise early but use it late in processing
Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise
Artificial IntelligenceComputation and Language
Summary
This paper looks at how transformer AI models understand how expert their conversation partner is during a dialogue. The authors found that while the model stores this information early in its internal layers, it doesn’t actually use it to shape its responses until much later. This means the model has some knowledge available before it influences output, and interventions to change behavior based on partner expertise must target deeper layers. The study shows this gap using dialogue examples involving different expertise levels.
What this means in practice
- •For chatbot developers: Design targeted interventions in transformer dialogue models by focusing on deeper layers where inferred partner expertise actively affects responses.
- •For conversational ai trainers: Improve training methods by monitoring and steering latent partner expertise representations before they change model behavior.
Tested on one dataset.
Authors
Mika Okamoto, Gabriele Sarti
Abstract
A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute does not yet influence the output. This gap between where information is readable and where it is used has been shown for attributes stated directly in the input. We ask whether it also holds for an attribute the model must infer gradually over a conversation, namely how expert its dialogue partner is. Using ExpertCollab, a corpus of multi-turn research-planning dialogues between model-played personas at four expertise levels, we find that partner expertise is most decodable in the early layers and falls to near chance before the midpoint of the network. Counterfactual patching shows that injecting the expertise difference at the layer of peak decodability barely changes a fixed late-layer readout, whereas the same difference injected past the midpoint propagates almost completely, a separation of more than an order of magnitude. A content-matched random control and a probe-free diagnostic place the transition at the same early layer, and a statically specified control attribute stays decodable throughout. An inferred relational attribute is therefore represented well before it becomes causally active, which bounds where any attempt to read out or steer partner-conditioned behavior must intervene. We use one model on a synthetic corpus as an initial demonstration.