Structure residual biases improve language model pretraining performance
Structure--Residual Inductive Biases for Language Model Pretraining
Computation and LanguageArtificial Intelligence
Summary
Predicting the next word helps AI models learn both to guess upcoming words and to build internal understandings. This paper explores whether explicitly separating the overall structure from detailed content in how models represent language can help. The authors propose two methods: one focuses on information at segment boundaries, and another splits representations into main structure and leftover details. They show that combining these methods improves language model accuracy beyond standard training alone.
What this means in practice
- •For natural language processing engineers: Improve language model training efficiency and accuracy by integrating structure-residual biases to better capture language structure and detailed information.
- •For speech recognition teams: Enhance pretrained language models used in speech recognition systems by applying complementary representation methods that better separate structure and fine details.
Authors
Hai Huang, Bartosz Żuk, Rassul Magauin, Jiaxin Huang, Xue Liu
Abstract
Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes representation geometry and induces a spectral decomposition into low-frequency structure and high-frequency residual. We derive coding bounds for both views and show that multiplicity-corrected boundary attention becomes exact as the within-segment path energy vanishes. Empirically, BBC and CC exhibit strikingly different depth preferences: BBC works best in early layers, whereas CC is most effective in later layers. At 300M training tokens, combining BBC and CC improves NLL by 1.44\% on GPT-2 Medium. These gains persist with training runs up to 20B tokens. Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. Together, these results suggest that structure--residual biases can improve language-model pretraining beyond next-token prediction. Code: https://github.com/haihuang-ml/structure-residual