Power law correlations persist without critical temperature in hierarchical language models

Power-Law Correlations Without Apparent Temperature-Tuned Criticality in Fixed-Tree Hierarchical Languages

Machine Learning

Summary

This paper looks at patterns in how computer models generate sequences of tokens, like words in a sentence. The authors investigate whether adjusting a temperature parameter causes a sharp change in the model’s long-range correlations, which would mean a phase transition like in physics. By studying a controlled toy model with a fixed hierarchical structure, they find that correlations always follow a smooth power-law decay and don’t show signs of a sudden shift to an ordered phase. They explain this with an analogy to a physics model and clarify that hierarchical structure alone can cause these effects without special tuning.

What this means in practice

  • For language model developers: Improve understanding of how temperature affects long-range correlations in autoregressive sampling of hierarchical token sequences.
  • For statistical physicists: Use the analogy to Ising models on trees to study phase-like behavior in complex probabilistic models with hierarchical dependencies.

Tested on simulated data.

Authors

Marcel Kühn, Bernd Rosenow

Abstract

Sampling temperature can substantially reshape autoregressive generation, but it is unclear when temperature changes in long-range correlations reflect a genuine phase transition rather than a smooth change in statistics. We study this question in a controlled analogue of temperature sampling: a finite-depth probabilistic context-free grammar on a fixed hierarchical tree, where exact next-token conditionals are computed by belief propagation and then rescaled by a softmax temperature before autoregressive sampling. We measure two-point mutual information between tokens as a function of separation. Across the tested temperatures and grammar ensembles, mutual information remains well described by a power-law envelope. Temperature changes the correlation magnitude and fitted exponent, but within the accessible system sizes we find no long-distance plateau characteristic of an ordered phase and no clear crossover to short-range exponential decay. An approximate mapping to an Ising model on a tree explains how exponential decay in tree distance becomes a power law in token distance, allowing power laws to persist without tuning to a special decoding temperature. This controlled baseline clarifies which temperature effects can arise from hierarchical structure alone, before adding training, approximation error, or neural architecture.
View PDFOpen arXiv