Summary
Large language models learn by adjusting many parameters during training and then use these adjustments to predict text during inference. This paper studies a special type of language model called Power Law Decoder Representation models and breaks down how their internal parts change and interact over time during training and prediction. The authors describe exact mathematical identities and how different factors like optimizer steps and data influence these dynamics. They also test these theories with experiments, finding some expected effects and ruling out certain simplified assumptions. Overall, it helps clarify how the detailed training process connects to the model’s behavior at inference time.
What this means in practice
- •For machine learning engineers: Improve understanding of parameter changes and optimizer effects to fine-tune large language models more effectively.
- •For ai system integrators: Use insights about training-inference behavior to better predict model performance and design inference pipelines.
A theory result. No direct application yet.
Abstract
This monograph develops a unified account of training and inference in Power Law Decoder Representation language models (PLDR-LLMs). Exact finite work identities decompose changes in the absolute energy of the row-centered learned map into parameter contributions, signed interactions, and numerical observation defects. Positive affine blocking retains restarts at the row-constant face, while the augmented AdamW state supplies the complete dynamical description. Predictive renormalization acts on the complete conditional training law for a single pass over distinct corpus target blocks, retaining optimizer memory, remaining data, schedule, and numerical policy. Autonomous reductions require closure; approximate reductions carry successor and emission errors. Finite-population covariance, matched physical clocks, matrix fluxes, and signed temporal energy connect row dynamics to model-wide observations. Absolute row collapse, relative row concentration, operator stabilization, and predictive accuracy are distinguished. Experiments reveal observer and optimizer dependence, reject the tested autonomous row-state candidates, and support finite conditional prediction and state-specific operator reduction. Independent single-pass families exhibit moving finite fluctuation regions without establishing a thermodynamic critical class. Conditional symmetry, head limits, covariance flows, and readout error budgets specify assumptions needed to transfer scaling laws to inference. The theory separates exact identities, conditional dynamical claims, and finite empirical findings, with proofs, selected formal checks, and compact numerical evidence.