Copy-paste language models improve efficiency and long-context performance
Copy-Paste Language Models
Computation and LanguageArtificial Intelligence
Summary
Transformers are machines that process language and can sometimes copy words from what they read. The authors show that by adding a special 'copy-paste' part to these machines, they become better at copying text exactly from their input. This helps the machines learn faster, work better on long texts, and improve when they look up information to answer questions. They tested their idea on models of different sizes and found it works well without slowing down the machine.
What this means in practice
- •For ai engineers: Train language models that better replicate copied text for improved accuracy in long document processing.
- •For search engine developers: Enhance retrieval-augmented language models to return copied input text more reliably and efficiently.
Authors
Nathan Godey, Yoav Artzi
Abstract
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.