Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
2026-07-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how linear transformers, a variant of transformer models designed to work faster with long contexts, learn from given information without updating parameters. They explain linear transformers as learning a connection between different sets of data examples and their expected outputs, and provide mathematical guarantees on how well this learning happens regardless of data size. Their work also highlights a balance between how simple or complex the input data and hidden features are. Using these insights, they suggest new ways to improve the design of activation functions and loss calculations in linear versions of large language models originally based on softmax attention.
transformerin-context learningsoftmax attentionlinear transformercontext lengthattention mechanismgeneralizationactivation functionloss functiondomain generalization
Authors
Peilin Liu, Ding-Xuan Zhou
Abstract
Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning. With richer context, transformers adapt more effectively to the current use case without any parameter updates. However, the quadratic computational and memory complexity with respect to context length significantly slows data processing in softmax transformers. Linear transformers were proposed to address this issue by reducing the complexity to linear dependence on context length, but the design and understanding of the feature mapping in linear attention, from a theoretical viewpoint, remain unclear. In this paper, we investigate the approximation and generalization abilities of linear transformers under a two-staged sampling process from domain generalization. We show that linear transformers perform in-context learning as learning a mapping from context distributions to response functions. A dimension-independent convergence rate is obtained for our generalization analysis, which also exhibits the tradeoff between the regularities of data distributions and latent features. Guided by our theoretical framework, we propose a new perspective on activation and loss design for linearizing pretrained softmax large language models.