Language model metamodel finds text linked to model signals better than search
Towards interpreting any residual stream direction by scaling activation inversion
Machine LearningArtificial Intelligence
Summary
When large language models process text, they contain hidden signals that relate to meaning or intent, but these signals are hard to interpret. The authors created a special kind of language model called a MAEM that can take one of these hidden signals and generate example text that strongly activates it, even beyond what exists in a given text corpus. This helps explain what a particular hidden signal represents by showing text that causes the model to react in that way. Their approach worked better than just searching for text or using previous methods, and it can even reveal concepts behind model behavior or hidden backdoors.
What this means in practice
- •For machine learning engineers: Identify and verify the concepts behind neural network directions for better model interpretability and debugging.
- •For nlp security teams: Detect and understand hidden backdoor attacks in language models more effectively than corpus search methods.
Authors
Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger, Tomáš Gavenčiak, Gavin Leech, Euan Ong, Jiahai Feng
Abstract
To interpret a direction in a language model's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to what that corpus contains. We develop Maximally-Activating Example Metamodels (MAEMs), language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions. Our objective is to maximise the cosine similarity of a supplied residual stream direction with the activations of the text generated by the MAEM. This label-free RL objective helps to scale MAEMs on diverse sources of residual stream directions. Empirically, we find the resultant metamodel produces informative exemplars: a Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder (NLA), beats corpus search (at 10M tokens) on a majority of held-out languages/domains, and activates 89% of held-out SAE features. It also identifies concepts behind steering vectors about as well as NLAs, recovers the payloads of LoRA backdoors more often than corpus search, and can even verbalise some model workspace inner monologue. Unlike explanations produced by other metamodels (e.g. NLAs), MAEM exemplars can be verified with a forward pass on the subject model, making the MAEM objective a grounded alternative for scaling the training of metamodels.