Weakening neurons appear late in transformers but shape output strongly
Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence
Machine LearningComputation and Language
Summary
Large language models have special parts called neurons that help decide what the model should say next. The authors found that some neurons actually weaken certain signals rather than strengthen them. These weakening neurons show up mostly in the later layers of the model but have a surprisingly big impact when they activate. This is unexpected since parts of the model thought to be inactive or unimportant actually play important roles.
What this means in practice
- •For large model engineers: Design and diagnose transformer models by identifying influential weakening neurons that affect output unexpectedly in later layers.
- •For ai safety teams: Monitor and control model behavior by targeting weakening neurons that activate often and impact outputs, improving interpretability.
Authors
Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui, Hinrich Schütze
Abstract
We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its input (reading) and output (writing) weight vectors. In this scheme, a strong negative cosine similarity indicates the neuron weakens the direction it detects in the residual stream, so we call this a weakening neuron. This allows us to gain a number of novel insights. First, we show that nine different LLMs have similar patterns: weakening neurons appear mostly in late layers whereas their counterparts, (conditional) strengthening neurons, are frequent in early-middle layers. Second, we find that weakening neurons display surprising behavior: even though there are few, they activate often and have a large influence on model behavior. Third, weakening neurons have a strong effect on model output when gate values are negative -- which is surprising since negative gate values are not expected to encode functionality.