Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Computation and Language
Summary
The authors found that even very advanced language models struggle with a basic task: perfectly copying a piece of text. They explain this problem comes from how these models understand the order of words using positional encodings, which aren't good at precisely matching exact positions. To fix this, the authors propose a new method called 2D-RoPE, which arranges text in a grid format instead of a line, making copying easier by using row and column positions. Their tests show this new approach lets simpler models copy much longer texts perfectly and also helps bigger models during large training. They suggest that treating text like a 2D grid could improve language modeling overall.
Authors
Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu
Abstract
While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.