Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
2026-07-20 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explore whether the popular transformer AI design is the best for every task or dataset. They create a method that tweaks key transformer parts using learned functions tailored to specific data, then test how well these tweaked transformers work on different tasks. They find some new transformer designs that learn faster and generalize better for simple tasks but are very task-specific. For language and code tasks, the improvements are smaller but more general across datasets. Overall, the authors show standard transformers are not always the best, and specialized designs can outperform them but may lose versatility.
TransformerInductive biasGeLUSoftmaxOut-of-distribution generalizationAlgorithmic tasksLanguage modelingCode modelingArchitecture optimization
Authors
Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey
Abstract
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be key for pushing AI beyond merely scaling current designs. *Method.* We present a method to optimize a transformer architecture for a given dataset, which we use as a tool to study optimal task-specific inductive biases. This method replaces the most important non-linearities (GeLUs,;softmax) with functions learned on held-out data. We then train the resulting architectures on other datasets, as a way to evaluate the compatibility between pairs of tasks. Findings. On algorithmic toy tasks, we identify new architectures with dramatic improvements in learning speed, in- and out-of-distribution generalization, and stability across seeds. The new designs prove very task-specific however, and indicate that these tasks require inductive biases very different from those of standard transformers. On code and language modeling datasets, we also find architectures with consistent, yet smaller improvements. These designs transfer much better across datasets and domains (English & computer code). Implications. Our results show that standard transformers are rarely a local optimum in the space of architectures. Simple alternatives can perform much better but sacrifice universality. This suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.