Language models store multiple timers separately in overlapping spaces
Factorized Representations of Concurrent Temporal Variables in Pretrained Language Models
Machine LearningComputation and Language
Summary
Sometimes language models need to keep track of different things that change independently, like when a topic changes or a language switches in a document. The authors studied how these different timing signals are stored inside the model's hidden layers. They found that these timers can be identified separately, even though their information overlaps in the model's internal representations. Changing one timer's representation affects only that timer’s information but not the other, showing a kind of organized structure in how models handle multiple evolving contexts.
What this means in practice
- •For natural language processing engineers: Improve model interpretability by isolating independently evolving context features like topic and language shifts.
- •For multilingual ai developers: Design language models that better handle multiple concurrent contexts such as language switching and topic changes.
Authors
Xinhe Zhang, Arnau Marin-Llobet
Abstract
Language models often need to maintain multiple contextual variables that evolve independently within a document. Prior work has identified structured representations of individual temporal or contextual quantities, but their joint organization remains less understood. We study this question using controlled documents with independently phased language-switch and topic-change timers. Across ten pretrained checkpoints, both timers are linearly decodable from the same hidden states. Removing a timer-associated centroid plane substantially reduces access to its associated variable while having little effect on the concurrent timer, although the planes are nonorthogonal and contain cross-timer information. Separate timer contributions also explain most reproducible variation among joint-state centroids. These signatures persist from 3 to 51 timer values and across Qwen3 checkpoints from 0.6B to 14B. Schedule position affects continuation probabilities, but the tested centroid displacement produces no consistent timer-specific output effect. Together, these results support approximate factorization based on selective accessibility and approximately additive joint-state geometry in overlapping subspaces.