Hidden traits in ai language models shift when channels are blocked
A balloon effect: ban one channel and a hidden trait finds another
Artificial Intelligence
Summary
Sometimes language models learn hidden preferences that don’t show up directly in their outputs. The authors found that blocking the usual ways these traits appear just moves them to other hidden parts, like squeezing a balloon and making it bulge elsewhere. By studying these hidden pathways inside the models, they identify specific parts in the model’s 'brain' that add these traits and show it is possible to block the traits more directly. This gives a better understanding of how unwanted influences get passed along and points toward ways to prevent them.
What this means in practice
- •For ai safety engineers: Prevent unwanted hidden traits in language models by targeting specific neural circuits that write those traits into outputs rather than only surface tokens.
- •For machine learning developers: Improve training processes for language models to reduce the transfer of subliminal biases embedded indirectly through outputs.
Authors
Keuntae Kim, Yong Suk Choi
Abstract
Language models trained on the outputs of other models can acquire traits that those outputs never mention, such as a preference for an animal carried by lists of numbers. Such subliminal learning can pass on unwanted traits. The intuitive defence is to find and forbid the tokens through which a trait shows, but how a trait's trace behaves under such bans, and what of it a student learns, has not been resolved. We closed a trait's outlets one by one in two stress tests, a preference chain and a name chain, across two models, six output formats and 16 traits. Bans did not remove the trait. Much as squeezing a balloon in one place makes it bulge in another, the trait moved to wherever the format still allowed, an effect we call the balloon effect. Decomposing this transfer, which bans on surface features cannot stop, down to single heads and neurons, we show that it can be cut at the circuit that writes the trait into the output. Patching that circuit while the teacher writes numbers stopped the chains, sharply reduced the overt name leak and halved the trait passed to a student, whereas the same number of random heads produced no detectable reduction. These results give a mechanistic account of how unwanted hidden traits are transferred and a route to preventing it.