Transformer models improve copying on repeated text with position hashing
Copying Under Repetition: From Prefix Matching to Position Hashing
Machine LearningComputation and Language
Summary
Copying long repeated texts is surprisingly tricky for AI models because the repeated parts look the same and confuse where to stop copying. The authors found a way to teach a special kind of transformer model to remember positions using math tricks with prime numbers, so it copies repeated text perfectly up to very long lengths. They also proved that common transformer methods can't handle ending copying correctly when there are too many repeated characters. Their method works well in experiments where other models fail.
What this means in practice
- •For natural language processing engineers: Create more reliable text generation systems that handle long repeated sequences without truncation or errors.
Authors
Anxin Guo, Tianlong Huang, Chenxiao Yang, Zhiyuan Li
Abstract
Frontier LLMs can copy random text almost flawlessly, yet they often overrun or truncate long blocks of repeated characters. In repeated strings, matching local content no longer reliably identifies where copying should continue or when it should stop. Nevertheless, we show that a two-layer transformer with $O(\log\log N)$-bit precision and $O(\log N/\log\log N)$ width can copy every string up to a chosen design horizon $N$ and terminate correctly under arbitrary repetition. We use RoPE-style rotations of attention values and outputs (VO-rotations) with small prime periods to encode distances from delimiter anchors modulo those primes. By the Chinese Remainder Theorem, these periodic encodings jointly distinguish all required distances, allowing one retrieval rule to select either the next source token or the end marker. In contrast, we prove a width-independent impossibility result for copying under sequential finite-precision softmax accumulation. It applies to transformers whose only positional input is an additive attention bias that eventually becomes constant, including NoPE and a recursive implementation of ALiBi. Once an identical-token run exceeds the resulting saturation threshold, further repetitions leave the logits unchanged. If this threshold grows sublinearly with $N$, these models cannot both complete a long all-identical copy and stop correctly, a pattern resembling the termination errors of frontier LLMs. In synthetic copying experiments, baseline transformer models fail on long repeated strings even within the training range. Trained VO models, by contrast, generalize beyond the training lengths with near-perfect accuracy up to length $\approx 5005$, the selected primes' period product.