Transformer models cut parameters using new dual-headed design
Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms
Machine Learning
Summary
Transformers are a powerful technology often used in AI but require lots of memory and parameters. This paper explores simplifying transformers by using two specialized parts instead of three in the attention mechanism. The authors tested different mathematical transformations to replace some components, making the model smaller and faster. Their approach worked better than the traditional three-part system on a large image recognition benchmark.
What this means in practice
- •For computer vision teams: Integrate dual-headed transformers with orthogonal transforms to reduce model size and improve image classification accuracy.
- •For embedded systems engineers: Use dual-headed transformer designs with efficient transforms to deploy AI models in memory-limited devices.
Authors
Ene Meco, Emadeldeen Hamdan, A. Enis Cetin
Abstract
To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (WHT), Discrete Cosine Transform (DCT), Discrete Fourier Transform, filterbank based Shearlet Transform, and Multiplication-Avoiding (MA) operators to construct dual heads. We combine spatial patches and their orthogonal transforms (or Shearlet and MA operators) in a structure similar to the attention block. We obtained better results than triple headed transformers in ImageNet. Extensive simulation examples are presented.