Transformers struggle to generalize token distances without positional cues
Distance generalization in transformers: why bother with positional encoding?
Computation and Language
Summary
Transformers are a type of AI model used for language and other tasks, and they rely on knowing the order of words or tokens to work well. This paper looks at how well transformers can handle changes in the distance between tokens, even when the overall input length stays the same. The authors test different ways of telling the model about position, like special positional codes, and find that these help the model handle new distances better. They also explore how seeing different distances during training affects the model’s learning and when learning new distances can help or hurt performance.
transformerspositional encodingdistance generalizationRoPEALiBicontext lengthdelay copy taskout-of-distribution generalizationtransfer learninginter-token distance
Authors
Daniel Henrik Nevermann, Claudius Gros
Abstract
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.