Transformers need multiple heads to check ordered bit sequences
Transformer Heads Looking for Order
Artificial IntelligenceMachine Learning
Summary
This paper looks at how transformers, a type of AI model, can tell if a string of bits (1s and 0s) is in order. The researchers found that if the transformer has just one attention head and one layer, it can't do this task. But if it has two attention heads in one layer, it can correctly check for order. This finding relies on the transformer having an output MLP (multi-layer perceptron), which helps with processing.
What this means in practice
- •For machine learning engineers: Design transformer models with multiple attention heads to improve ability to detect ordered structures in data sequences.
- •For hardware accelerator designers: Evaluate hardware needs for supporting multi-head attention efficiently to enable more complex sequence recognition.
A theory result. No direct application yet.
Authors
Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser, Tomasz Steifer, Felipe Urrutia, José Verschae, Przemysław Andrzej Wałȩga
Abstract
In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.