Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
2026-09-03 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how large language models learn during training and found that seeing different versions of the same information (called auxiliary views) helps learning. They showed that repeating information is important, but when keeping the total amount of data fixed, using these alternative views is actually better than just repeating the same text. This improvement happens regardless of how good the examples are and helps especially with certain kinds of knowledge. Their work also explores how these effects show up inside the model's layers, suggesting that having diverse ways to see knowledge is important for training success.
Large Language ModelsPre-trainingAuxiliary ViewsParaphrasingToken BudgetFactual RecallTeacher ModelContextual KnowledgeFoundational KnowledgeLayer-wise Bias
Authors
Joseph Lee, Yidi Huang, Dokyoon Kim, Shu Yang, Li Shen
Abstract
Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and clarify that paraphrasing helps only at smaller batch sizes. Second, holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning, counterintuitively, even for factual recall. Third, the effectiveness of auxiliary views is not contingent on the strength of the teacher model that generates them. Fourth, we identify forms of knowledge, contextual and foundational, that aid learning in the presence of prior knowledge gaps. Finally, we examine how these effects manifest mechanistically via layer-wise biases and compression. Together, our findings suggest that auxiliary representations of knowledge, which arise naturally in large pre-training corpora, are a key factor in the success of pre-training and offer a plausible explanation for why data diversity matters.