Residual streams and recurrent states reveal language model memory patterns
Residual Streams Read, Recurrent States Remember: The Global Workspace in Mamba Models
Artificial Intelligence
Summary
Transformers and state-space models process language differently, and this paper looks at how memory and information flow inside these models compare. The authors studied Mamba models and found that combining two parts of the model, the residual streams and recurrent states, helps better identify what the model 'knows' at different steps. They show that the recurrent state often carries information longer than the residual stream, which can disappear faster. This work points out that these two components play distinct but complementary roles in how the model handles information.
What this means in practice
- •For ai developers: Improve interpretability tools by jointly analyzing residual and recurrent state readings in state-space language models.
- •For language model engineers: Design better state-aware interventions in models by leveraging recurrent state persistence for targeted content redirection.
Authors
Wenlong Wang, Fergal Reid
Abstract
Can the global-workspace account of transformer representations extend to state-space language models? We fit Jacobian lenses to the residual streams and recurrent states of Mamba-1, Mamba-2 and Mamba-3, using the original 1000-prompt recipe. Joint residual--state readouts improve recovery of known intermediate concepts over the residual lens on at least five of six task families in every tested Mamba checkpoint. On Mamba-2, state alone exceeds residual and logit lenses on all six families; a normalised joint readout improves on both components on five. Temporal maps and word-list experiments show earlier content remaining state-readable as residual visibility changes. We also propose sign-guarded steering, which improves target top-five success over coordinate exchange on matched verbal-report trials in five models. Recurrent state alone supports this verbal access. These gains do not extend consistently to relational answers: guarded edits often output the edited concept itself, and Mamba-3's joint edits can disrupt successful state-only redirection. Recurrent state thus provides a complementary carrier of workspace content, whose recovery, persistence and causal uses require separate measurements.