Researchers map detailed internal features for time aware recall in language models
Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall
Computation and LanguageMachine Learning
Summary
Language models sometimes get facts wrong because they mix up info from different times. The authors found specific tiny parts inside these models that help them remember facts correctly based on time. They looked at different models and saw that remembering time happens in a complicated way, not just step by step. Their work can help improve how language models keep up with facts that change over time.
What this means in practice
- •For ai developers: Target identified MLP components to improve factual accuracy for date-related queries in language models.
- •For nlp engineers: Design model debugging tools that isolate and analyze time-sensitive recall features for better model interpretability.
Authors
Sanjay Govindan, Yang Song, Maurice Pagnucco
Abstract
Large Language Models (LLMs) suffer from temporal misalignment, often due to the contradictory nature of their training corpora. While current mitigation strategies rely on computationally expensive fine-tuning or context-heavy retrieval augmented generation (RAG), the internal mechanisms governing time-sensitive recall remain under-explored. Unlike prior studies that identify temporal components such as attention heads and MLP layers, we provide the first feature-level map of temporal recall by isolating individual MLP features via transcoder circuit tracing. We identify three node categories (common temporal, common to the year, and chrono-semantic) which interact to generate a temporal filter during factual recall. By analysing Gemma 2 2B, LLaMA 3.2 1B, and Qwen3-4B, we show that these features do not follow a simple linear pipeline but represent time through a parallel and mixed syntactic-semantic interplay across layers. We additionally discover a class of higher-layer temporal components invisible to existing EAP-IG methods, establishing transcoders as a more complete lens for temporal interpretability in time-sensitive factual recall. These findings present MLP components for potential targeted interventions in time-sensitive factual recall