Transformers learn multiple tasks efficiently by sharing common patterns
Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
Machine Learning
Summary
Transformers, a type of AI model, can quickly learn new tasks with only a small example prompt because they recognize common structures shared across many tasks. The authors study how these shared patterns reduce the amount of data needed for learning new tasks in context. They provide a mathematical way to measure how complex the space of related tasks is and build a Transformer model that mimics this approach. Their results explain why training on many related tasks helps Transformers adapt better to new ones with limited examples.
What this means in practice
- •For machine learning engineers: Design pretrained Transformer models that efficiently adapt to new tasks by exploiting shared structures among tasks to reduce training requirements.
- •For software developers building ai tools: Create AI systems capable of adapting to various user requests quickly using limited examples by leveraging cross-task similarities explained in this work.
A theory result. No direct application yet.
Authors
Zhongjie Shi, Rongjie Lai, Alexander Cloninger, Wenjing Liao
Abstract
Transformers achieve remarkable performance by jointly learning broad families of tasks during pretraining and adapting to unseen tasks from only a short prompt. Yet a rigorous mathematical and statistical understanding of this phenomenon remains limited. This paper aims to study how Transformers exploit shared cross-task structure and how this structure affects the sample complexity of in-context learning (ICL). Specifically, we characterize task-space complexity through covering numbers under a prescribed metric, thereby quantifying the low-dimensional cross-task structure without requiring an explicit parametric representation. The resulting cover provides a set of anchor functions, which we use to introduce a task-identification-and-evaluation procedure: context observations localize an unseen task among the anchor functions, and the response at a query is predicted by aggregating the corresponding anchor function query evaluations. For approximation, we explicitly construct a Transformer with Softmax attention to approximate this procedure. For generalization, we derive an error bound that separates the effects of the number of pretraining tasks and the prompt length. The scaling with respect to the number of pretraining tasks is governed by the intrinsic dimensions of the task space and input domain; once sufficiently many tasks are available, the dependence on the prompt context length becomes dimension-free. To the best of our knowledge, this is the first work to quantify cross-task complexity for general nonlinear task families and explicitly construct a Transformer that exploits their low-dimensional structure to perform ICL. Our theory provides a quantitative explanation of how joint pretraining across related tasks improves in-context generalization.