Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models
Machine LearningComputation and Language
Summary
The authors improve on sparse autoencoders, which usually only analyze each word in a sentence by itself, by making them remember some information across several words. Their method, called Persistent SAEs, decides how long certain features should stick around while processing a sentence. They found that fast features detect immediate details, while slow features capture overall topics and remain useful over long stretches of text. This helps with understanding and monitoring language models better, especially for spotting certain patterns throughout a conversation.
Authors
Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik
Abstract
Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence. We introduce Persistent Sparse Autoencoders (Persistent SAEs), which extend standard SAEs by learning a persistence coefficient for each feature, allowing the model to learn which features should persist and for how long. Our experiments show that they retain competitive reconstruction quality while learning a spectrum of feature timescales: fast features behave as locally interpretable detectors, whereas slow features concentrate topic-level information in a persistent state. Moreover, as shown in a prompt-injection monitoring case study, slow features preserve detection signals and remain causally effective over long contexts. These results suggest that Persistent SAEs open up new opportunities for interpreting and monitoring language models through persistent semantic representations.