Transformer models generalize better with fewer long training strings
Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones
Machine LearningArtificial IntelligenceComputation and Language
Summary
Transformer language models can handle longer and harder inputs than those they were trained on, but understanding why is tricky. This paper shows how training on fewer, longer sequences can lead to better learning compared to many shorter ones. The authors provide mathematical guarantees about this behavior and explain how model size, input length, and training data interact. Their experiments support these findings.
What this means in practice
- •For natural language processing engineers: Design training regimens that use longer text samples to improve transformer model generalization efficiency.
- •For bioinformatics software developers: Optimize models analyzing genetic sequences by balancing training sample length and quantity based on the paper’s generalization bounds.
Authors
Marius Henry, Michael Rizvi-Martel, Michael Hahn
Abstract
Transformer-based language models exhibit an astonishing capacity to generalize to longer and more difficult problem instances than those seen in training. However, theoretical understanding of this phenomenon still lags behind. Unlike most machine learning models, the generalisation abilities of transformers inherently require considering two axes. On one hand, one must examine in-distribution statistical generalisation, which concerns the ability of a model to broaden a function from training samples to novel samples of the same distribution. On the other hand, given the sequential nature of language data, one must also consider length generalisation, which reflects whether a model trained on inputs of a given length can extrapolate to longer ones.These two notions have mostly been studied separately, leaving open how they interact. In this work, we give a joint treatment of both axes for fixed-precision transformers. We give generalisation guarantees in terms of the parameter sizes of the involved models, as well as the quantization space and the alphabet size that supply the sequences. Finally, we illustrate how the requirements in the sample count and training length trade off. In particular, we find that increasing training length can be exponentially more efficient than increasing the sample size. Experiments support predictions of the theory.