Hybrid language models organize memory circuits through token positioning
The Token Before the Value Is the Key: How Hybrid Architectures Organize Induction Circuits
Machine Learning
Summary
Understanding how language models remember and use past words helps improve their design. This paper shows that the token immediately before a repeated value plays a key role in how the model copies and matches information. The authors studied hybrid models, tracking how different layers handle memory tasks, and found that certain layers are specialized for carrying past tokens while others match or copy them. Changing how the model processes nearby tokens can shift these roles, affecting its ability to recall text.
What this means in practice
- •For machine learning engineers: Design hybrid language models that allocate memory tasks efficiently across layers to improve text recall and model efficiency.
- •For natural language processing developers: Tune model architectures by adjusting local token windows and training data to influence the development of memory circuits for better text generation.
Authors
Ke Cheng, Xin Xu, Yixiao Chen, Lei Xin, Jianbo Zhao, Fanhu Zeng, Yue Liu, Jun Zhang, Jie Jiang
Abstract
Hybrid language models can improve capability as well as efficiency, raising the question of how architectural complementarity becomes learned computation. We examine the established induction roles of Carrying predecessor information, Matching a source by content, and Copying its value. How are these position-sensitive and content-based computations allocated across heterogeneous layers? We introduce layer-type-agnostic paired probes that track Carrying and Matching through a common block-update interface. In recurrent--global and local--global hybrids, Carrying concentrates in efficient layers and Matching in global receivers. The measured local contribution concentrates on lag one: the token immediately before the historical value. Changing predecessor support through lag-one masking, convolution removal, or early learning-rate reduction can relocate Carrying and Matching between stages. Source-key restoration and fixed-value selection trace the receiver's dependence on the prepared source. These interventions also change natural-text recall, with outcomes depending on configuration and target. Varying local windows and induction-enriched training text changes the early development of functional Carrying and Matching, connecting architectural priors and training evidence to formation timing. Together, the probes and interventions shift the explanatory focus upstream: the organization of Matching follows how Carrying is learned. The token before the value provides a concrete link between a hybrid's architecture, circuit development, and recall. Code is available in https://github.com/ckpassenger/bind-match-copy/tree/main.