Test-Time Registers as Global Priors for Tokenized Image Generation
Computer Vision and Pattern Recognition
Summary
The authors studied a phenomenon in attention-based vision models where certain tokens, called registers, repeatedly attract attention and relate to broad, low-detail image patterns. They found that these registers carry global information that can be extracted and used without retraining the model. By creating a method named RegToken, they turned this register information into special tokens that help improve image generation quality and speed up optimization. Their work shows that what was once seen as a quirk in the model’s attention can actually be used as useful global signals in generating images.
Authors
Cheng-Yao Hong, Yifan Wang, Yuewei Lin, Chenyu You
Abstract
Attention-based models often develop attention sinks, where a small number of tokens repeatedly attract attention and accumulate unusually large activations. In vision transformers, these outliers are closely related to registers, which have been diagnostically linked to global, low-frequency image structure. Existing work has largely studied registers through interpretability analyses and linear probes, leaving open whether they can be operationalized as plug-and-play signals for generation without retraining. We revisit this question in tokenized image generation. Using OpenCLIP and DINOv2 on ImageNet, we find that test-time register features exhibit stronger low-frequency concentration than both [CLS] readouts and patch-mean features, and show a consistent (albeit moderate) correlation with pixel-space DCT low-frequency energy. Motivated by these diagnostics, we introduce RegToken, a training-free procedure that converts register structure into a small set of global prior tokens by (i) NFN-based layer localization, (ii) TokenRank-guided subspace extraction, and (iii) a projection-and-conservation update on the register subspace. Inserted into a frozen compact 1D token generation pipeline, RegToken improves ImageNet generation and alignment metrics (e.g., FID-5k 20.5 to 20.1, SigLIP 3.6 to 3.9) without modifying pretrained weights, and accelerates test-time optimization (Steps@$τ$ 74 to 52). Overall, our results suggest that structures often viewed as attention artifacts can be repurposed as lightweight global priors for tokenized generation.