Data and memory needs linked by one energy spectrum in autoregressive models
One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction
Machine LearningArtificial IntelligenceComputation and Language
Summary
The paper studies how much memory a learned model needs to make good use of more data when predicting sequences. The authors show that both data size and memory size are controlled by one spectrum called the predictive-energy spectrum, which summarizes how information is distributed. They prove a mathematical rule that relates the prediction error to both the amount of data and the amount of memory, explaining how improving one or the other helps prediction. They also build a specific attention-based model that follows this rule and test ways to manage memory precision and computation.
What this means in practice
- •For machine learning engineers: Optimize memory usage and data scaling in sequence prediction models using the demonstrated data-memory energy tradeoff.
- •For hardware architects: Design memory and compute precision for autoregressive models informed by the paper’s compute-precision tradeoff results.
A theory result. No direct application yet.
Authors
Chiwun Yang, Xiaoyu Li
Abstract
How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing $μ$ for the resulting energy spectrum, we prove the minimax law $\mathfrak R^*_{\rm value}(n,B)\asymp_R Φ_μ(n^{-1})+Φ_μ(τ_B), Φ_μ(t)=\int\min\{x,t\}\,μ(\mathrm dx),$ for $n$ prediction blocks and a learned state with at most $2^B$ values. Data set the resolution $1/n$; memory sets the level $τ_B$ reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.