Vision transformer performance improved by convolutional self-attention network
ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision Transformers are a popular type of deep learning model for analyzing images, but they usually need very large datasets and a lot of computing power. The authors propose ACSAN, a model that replaces some heavy parts of Vision Transformers with convolution operations, which are simpler and more efficient. They created a special module called Dynamic Convolutional Self-Attention that combines different convolution techniques to better understand image features. Testing ACSAN on popular image datasets shows it performs as well or better than Vision Transformers but with fewer parameters, making it more efficient.
What this means in practice
- •For mobile app developers: Create image classification apps that perform well on limited hardware by using ACSAN’s efficient attention mechanism that reduces parameter count.
- •For embedded systems engineers: Design computer vision models for devices with limited training data and computing resources by substituting Vision Transformer components with ACSAN’s convolutional modules.
Authors
Zimian Wei, Tianqi Zhang, Hengyue Pan
Abstract
In the past few years, Vision Transformer (ViT) has achieved excellent performance in the field of computer vision. However, there still some problems remain. For instance, ViTs heavily rely on large-scale training data since they simply contain a large number of trainable parameters, which slows down their learning efficiency and limits their applicability in resource-constrained scenarios. In this paper, we propose a novel deep learning model, namely All-Convolutional Self-Attention Network (ACSAN), which applies convolution operations to replace fully-connected matrices multiplications in the ViT framework. To implement ACSAN, we firstly design Dynamic Convolutional Self-Attention (DCSA) as the basic module, which employs multi-scale dilated convolutions to generate Query, Key, and Value. We defined spatial self-attention and channel self-attention to generate attention features respectively. The two attention features are finally merged as the output of the DCSA module. Based on DCSA, we then construct the basic ACSAN block. Comparing with ViT, we use DCSA to replace the regular self-attention operations, and use convolution layers to replace MLPs. To evaluate the performance of the proposed method, we train ACSAN on three widely-used image classification databases, i.e. CIFAR-10, CIFAR-100 and ImageNet-1k, from scratch. Experimental results show that ACSAN achieves excellent classification accuracy comparing with standard ViT with comparable or fewer parameters, as well as other state-of-the-art methods. Ablation studies verify that each component contributes positively to the final performance.